PaperlayerView exact revision on arXiv

Source-linked AI summary

COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training

Alex Jinpeng Wang, Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Kevin Lin, Zhengyuan Yang, Lijuan Wang, Mike Zheng Shou

arXiv:2401.00849v1cs.CV

TL;DR

Long-context vision-language models support few-shot text generation but face alignment challenges, and high-quality interleaved video-text data remains limited. The paper presents CosMo, which adds contrastive learning, and Howto-Interlink7M, an interleaved video-text dataset. CosMo outperforms Open-Flamingo across image-text and video-text benchmarks while using fewer samples.

  • Problem

    Autoregressive vision-language models support long-context few-shot generation but face alignment challenges, while interleaved video-text data capturing relationships across clips remains deficient.

    Source: §1 · p. 2

  • Method

    CosMo partitions an LLM for unimodal text processing and multimodal fusion, adds contrastive loss, and is paired with Howto-Interlink7M, a detailed interleaved video-text dataset.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Across 14 image-text and video-text benchmarks, CosMo achieves superior performance to Open-Flamingo [3] using fewer samples; it has 34% learnable parameters.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that Howto-Interlink7M’s high-quality video-text data improves CosMo’s performance on both video-text and image-text tasks.

    Source: §1 · p. 2

    MSVD and MSRVTT ground truths are compromised because they were generated by rule-based methods rather than human annotation.

    Source: §6 · p. 7

Abstract

from arXiv · show

In the evolution of Vision-Language Pre-training, shifting from short-text comprehension to encompassing extended textual contexts is pivotal. Recent autoregressive vision-language models like \cite{flamingo, palme}, leveraging the long-context capability of Large Language Models, have excelled in few-shot text generation tasks but face challenges in alignment tasks. Addressing this gap, we introduce the contrastive loss into text generation models, presenting the COntrastive-Streamlined MultimOdal framework (\ModelName), strategically partitioning the language model into dedicated unimodal text processing and adept multimodal data handling components. \ModelName, our unified framework, merges unimodal and multimodal elements, enhancing model performance for tasks involving textual and visual data while notably reducing learnable parameters. However, these models demand extensive long-text datasets, yet the availability of high-quality long-text video datasets remains limited. To bridge this gap, this work introduces \VideoDatasetName, an inaugural interleaved video-text dataset featuring comprehensive captions, marking a significant step forward. Demonstrating its impact, we illustrate how \VideoDatasetName{} enhances model performance in image-text tasks. With 34% learnable parameters and utilizing 72\% of the available data, our model demonstrates significant superiority over OpenFlamingo~\cite{openflamingo}. For instance, in the 4-shot flickr captioning task, performance notably improves from 57.2% to 65.\%. The contributions of \ModelName{} and \VideoDatasetName{} are underscored by notable performance gains across 14 diverse downstream datasets encompassing both image-text and video-text tasks.

1. Introduction

Long-context multimodal models can handle few-shot generation, but Flamingo’s text-generation design is not suited to classification, and high-quality interleaved video-text data remains scarce. The paper introduces CosMo and Howto-Interlink7M to address these model and data gaps.

1. Introduction

  • Flamingo [2] and PALM-E [14] use in-context learning to handle multiple image-text pairs and long inputs, supporting downstream tasks without fine-tuning.Flamingo’s text-generation-focused architecture is not optimally suited to classification tasks.Source: §1 · pp. 1–2
  • High-quality interleaved video-text data that captures relationships across clips remains deficient, despite long-caption and chapter-based video efforts.Public datasets are largely short image-text pairs; web documents can be noisy because their images may not be highly correlated.Source: §1 · p. 2
  • CosMo processes four input types, including interleaved data, and Howto-Interlink7M improves performance across 14 image-text and video-text benchmarks versus Open-Flamingo [3] using fewer samples.The authors also report that high-quality video-text data from Howto-Interlink7M helps on image-text tasks.Source: §1 · p. 2
  • With 34% learnable parameters, CosMo is reported to outperform [3] and uses an additional contrastive loss for interleaved-data pre-training.The paper also presents Howto-Interlink7M and reports that interleaved video-text data boosts performance on image-text and video-text tasks.Source: §1 · p. 2

2. Related Work

Prior vision-language work moved from short image-text pairs toward long-form and interleaved inputs, but contrastive models retain an advantage on classification and video interleaving remains underexplored. CosMo combines Flamingo’s input design with contrastive learning, alongside a new interleaved video-text dataset.

2. Related Work

  • Flamingo [2] handles interleaved image/video-text data but falls behind contrastive models on classification, while CLIP [39] and CoCa [59] excel at contrastive learning.The related work describes Flamingo as scaling from a 1.4B to a 70B parameter language model.Source: §2 · p. 2
  • CosMo adopts Flamingo’s input design and adds contrastive loss within middle-level LLM representations to enhance visual-text alignment.The framework is intended to support both classification and generation tasks while streamlining learnable parameters.Source: §2 · p. 2
  • Existing interleaved datasets include web-document collections such as MMC4 [63] and Obelics, while MMC4 lacks exact image placement in documents.The paper introduces interleaved video-text data as a departure from work focused on noisy web documents.Source: §2 · p. 2

3. Howto-Interlink7M

Howto-Interlink7M is constructed from HowTo100M videos segmented into shots, with annotations designed to preserve detail and continuity across clips. The paper reports longer captions with improved video alignment compared with the original dataset.

3. Howto-Interlink7M

  • Figure 2 contrasts brief ASR captions in conventional datasets with shot-level Howto-Interlink7M annotations combining ASR, captions, and dense captions.The annotations use historical context and highlight object labels and connectives between clips.Source: Figure 2 · p. 3
  • Table 1 reports Howto-Interlink7M data statistics, including average token length and CLIP [39] image-text similarity for each clip in its last two columns.Source: Table 1 · p. 3
  • Howto-Interlink7M addresses weak ASR-to-video alignment by providing high-quality interleaved video-text annotations.In HowTo100M, only 51% of clips visibly depict objects or actions mentioned in their captions.Source: §3 · p. 3
  • The pipeline segments HowTo100M videos into shots, uses BLIP2 [28] for concise descriptions and GRIT [54] for dense region captions, then conditions later GPT-4 [35] captions on preceding clips.For the first clip, GPT-4 generates a summary from detailed annotations; instructions preserve ASR information, including nouns and actions.Source: §3 · p. 3
  • Table 2 presents CosMo architecture details and defines #CE as the number of Cross Attention layers.Source: Table 2 · p. 3
  • Howto-Interlink7M captions are longer and show higher CLIP similarity to video content than captions in the original HowTo100M dataset.The analysis also notes that specific details, including actor names and objects such as food, are preserved across clips.Source: §3 · p. 3

4. CosMo

CosMo combines language modeling and contrastive objectives in a streamlined LLM architecture for image, video, and interleaved inputs. Its preprocessing and data-sampling strategies address noisy matches and redundancy in training data.

4. CosMo

  • CosMo adds contrastive loss to Flamingo baselines and partitions the LLM into unimodal text-processing and multimodal data-handling components.The framework supports classification and generation while reducing learnable parameters.Source: §4 · p. 3

4.1. Overall Architecture

  • CosMo represents image-text, video-text, and interleaved variants as document-style sequences, then uses gated cross-attention for visual conditioning.Bottlenecks in feature channels reduce learnable parameters; cross-attention layers are placed every 2 blocks for CosMo-2B and every 4 for CosMo-3.4B.Source: §4.1 · p. 4, Figure 3 · p. 4

4.2. Enhancing Image-Text Alignment

  • A learnable query attends globally to visual tokens, and a projection head maps visual and text embeddings to the same dimensionality for contrastive training.The approach is intended to capture information across the full token set.Source: §4.2 · p. 4

4.3. Interleaved Data Preprocessing

  • For MMC4, low-similarity image-text matches can cause gradient explosions; CosMo perturbs alignment scores for matching and replaces texts for scores below 0.20 with pseudo-captions.The score disturbance uses a normal distribution with standard deviation in (−0.04, 0.04), clamped to −0.08 and 0.08; optimal transport then matches images to texts within a document.Source: §4.3 · p. 5

4.4. Training Details

  • Training data are drawn from a refined subset of commonly used image-text pretraining datasets, with clustering and filtering used to extract the subset.Figure 5 describes clustering and uniform distance-based sampling as ways to alleviate redundancy in LAION400M and similar large datasets.Source: §4.4 · p. 5, Figure 5 · p. 5

5. Experiments

CosMo is evaluated on image-text and video-text benchmarks, with ablations examining data selection, loss, sampling, and model configuration. It outperforms Open-Flamingo across nearly all image-text benchmarks, and Howto-Interlink7M further improves results.

5.2. Ablation Study

  • Contrastive loss improves performance over the baseline, but aggregating losses across nodes adds no commensurate benefit and slows iterations from 4.9s to 6.1s.The default applies contrastive loss over each GPU's batch.Source: §5.2 · p. 6
  • Removing interleaved image-text data drops the average score from 38.0 to 20.1; omitting conventional image-text pairs also hurts, and video-text removal harms most downstream tasks.The ablations use an 18M subset containing 12M interleaved, 4M image, and 2M video-text samples.Source: §5.2 · p. 6
  • The “minimum” dataloader strategy outperforms “maximum” and “round-robin,” supporting balanced quantities across training-data types.Gradient accumulation steps are matched to the number of data types so each iteration includes all types.Source: §5.2 · p. 6
  • Larger visual encoders tend to improve results slightly, while increasing parameters and iteration time from 1.2 seconds to 2 seconds.The authors choose the larger encoder as the default despite this cost.Source: §5.2 · p. 6
  • Reducing learnable parameters does not generally reduce performance; a layer interval of 2 and compression ratio of 2 even enhance it and are used in the final framework.The lightweighting approach reduces cross-attention layers and compresses their associated parameters.Source: §5.2 · p. 6
  • Longer interleaved sequences generally improve results across the tested range of 64 to 192, but increase computational demands and can slow training through GPU memory use.The setup uses sequence length 128 to expedite training under a limited computation budget; longer sequences may improve the model if budget allows.Source: §5.2 · p. 7

5.3. Few-shot Evaluation on Vision-Language Tasks

  • CosMo outperforms Open-Flamingo across nearly all image-text benchmarks while using fewer samples (130M vs. 180M) and fewer parameters.With the same RedPajama-3B language model, CosMo also substantially outperforms Open-Flamingo.Source: §5.3 · p. 7, Table 5 · p. 7
  • Adding Howto-Interlink7M improves results, despite pre-training with only three frames and evaluation at up to 32 shots.The experiments evaluate image-text and video-text benchmarks for few-shot in-context learning.Source: §5 · p. 5, §5.3 · p. 7

6. Training Details

The authors report robust performance across most evaluated downstream video-text tasks, with additional video data consistently improving results. Their analysis also examines interleaved-data similarity thresholds, sample composition, and zero-shot alignment evaluation.

6. Training Details

  • Adding a video dataset consistently enhances model performance across video captioning and video question-answering tasks.Except for YouCook2, evaluation uses three frames per video; parentheses in Table 6 indicate text-similarity evaluation.Source: §6 · p. 7, Table 6 · p. 8
  • MSVD and MSRVTT ground truths are rule-generated rather than human-annotated, limiting evaluation when detailed model predictions do not match reference wording.The authors describe an alternative VQA evaluation using text similarity from a pre-trained language model.Source: §6 · pp. 7–8

6.1. Analysis on Interleaved Data

  • Most similarity scores fall between 0.2 and 0.4; the authors compare thresholds from 0.18 to 0.30 on an 18-million-sample subset.Figure 6 shows clip image-to-text similarity score distributions for all pairs and matched pairs.Source: §6.1 · p. 8, Figure 6 · p. 8
  • Similarity thresholds below 0.22 trigger frequent gradient explosions, while thresholds of 0.28 and 0.30 yield suboptimal COCO and FLICKR30K results, partly after interleaved-sample dropout.Replacing noisy captions with CosMo-generated captions stabilizes training; replacing accurate pairs significantly decreases performance.Source: §6.1 · p. 8, Table 7 · p. 8
  • Replacing captions for exceptionally low-similarity images, about 5% of the dataset, yields a marginal but discernible overall performance enhancement.The stated aim is to preserve the maximum number of frames while addressing potential bias.Source: §6.1 · p. 8
  • Among three MMC4 subsets, the subset with the most frames shows a significant performance gap in its favor; the authors associate few-shot effectiveness with interleaved data and LLM use.The comparison uses randomly sampled, most-frames, and single-frame subsets, with training limited to one epoch.Source: §6.1 · pp. 8–9, Table 8 · p. 8

6.2. Zero-shot Alignment Tasks

  • The zero-shot alignment evaluation covers image classification and retrieval across 38 datasets without training.The evaluation follows the DataComp pipeline.Source: §6.2 · p. 9, Table 9 · p. 9

7. Conclusion and Limitations

The conclusion presents CosMo as an autoregressive multimodal model incorporating contrastive loss for in-context learning, alongside a new interleaved video-text dataset. It identifies further exploration of pretraining models on long-text tasks as future work.

7. Conclusion and Limitations

  • CosMo incorporates contrastive loss into an autoregressive multimodal model for in-context learning, while Howto-Interlink7M adds an interleaved video-text dataset with comprehensive captions.The authors say further exploration of pretraining models on downstream, particularly long-text, tasks is warranted.Source: §7 · p. 9

A.1. Data and Comparison

The appendix reports that CosMo outperforms Open-Flamingo with equivalent-scale pretraining data and describes training details, video multiple-choice evaluation, and extended ablations. The extended results favor interleaved data weighting and find negative gains from adding LoRA to cross-attention-based models.

A.1. Data and Comparison

  • With equivalent-scale pretraining data, CosMo consistently outperforms Open-Flamingo across diverse tasks.The comparison uses a larger Mistral-7B language model, for an overall parameter count of 8.1B.Source: §A.1 · p. 12

B.1. Hyperparameters

  • The training setup uses DeepSpeed ZeRO stage 2 with fp16 and aligns gradient accumulation steps with the number of data types; larger models require smaller batches under GPU memory constraints.The hyperparameter table covers three CosMo variations.Source: §B.1 · p. 12, Table 12 · p. 13

B.2. Handling Exceptions

  • The authors say their exception-handling strategies mitigate multi-node training challenges and support a more stable, reliable process.The strategies address unstable batches, excessive losses, and unrecoverable NAN errors.Source: §B.2 · p. 12

C. Video-based Multiple-choice Tasks

D. Extended Results: Multi-shot Details

D.1. Data Weight Ablation

  • Interleaved image-text pairs notably influence downstream accuracy, leading the authors to set their default interleaved-data weight to 2.The ablation compares weights across image-text, interleaved image-text, and video-text data.Source: §D.1 · p. 13, Table 13 · p. 13

D.2. PEFT method vs. Cross Attention

  • Adding LoRA to the text decoder while retaining cross-attention yields negative gains for cross-attention-based models that already have many trainable parameters.The comparison concerns parameter-efficient fine-tuning versus cross-attention.Source: §D.2 · p. 13

D.3. Learning Rate Ablation

The ablations show that very small learning rates hurt downstream accuracy and cosine scheduling performs best in most cases. For image-text pretraining, interleaved data had greater downstream impact than conventional image-text datasets.

D.3. Learning Rate Ablation

  • Accuracy on 8 shots was observed to drop from 40.7 to 21.9; default learning rates were 5e-4 for CosMo-2B and CosMo-3.4B, and 3e-4 for CosMo-8.1B.The experiment varied learning rates from 3e-4 to 3e-6, and extremely small learning rates significantly impaired downstream task performance.Source: §D.3 · pp. 13–14

D.4. Learning Rate Schedule

  • Cosine scheduling yielded the best results across most cases and was adopted as the default scheduler.Source: §D.4 · p. 14

E.1. Data Selection Strategy for Image-Text

  • Common image-text datasets had less downstream impact than interleaved data; using all 130M DataComp examples changed performance minimally, and COCO and Visual Genome were excluded to avoid overlap.Source: §E.1 · p. 14

E.2. Exploring Longer Sequences with Memory Bank

The memory-bank approach explored longer interleaved sequences, but higher computational demands and limited downstream gains led to its exclusion from the final model configuration.

E.2. Exploring Longer Sequences with Memory Bank

  • The experiment followed Memorizing Transformer principles to lengthen interleaved data; that approach is described as handling texts beyond 10K tokens.Source: §E.2 · p. 14
  • Longer-sequence experiments improved validation accuracy (63.012 to 60.85) but substantially increased computational demands and yielded restricted, sometimes negative, downstream gains.Source: §E.2 · p. 15
  • The Memorizing Transformer approach was not included in the final configuration after these findings, though alternative ways to handle extended sequences remain open for exploration.Source: §E.2 · p. 15

F. Training Progression

Training curves show lower language-modeling loss for larger models, especially on interleaved image-text data. Contrastive-loss convergence was slower under smaller batch sizes, with hardware limits constraining the largest model's batch size.

F. Training Progression

Related papers

Loading 2401.00849v1…