PaperlayerView exact revision on arXiv

Source-linked AI summary

COSA: Concatenated Sample Pretrained Vision-Language Foundation Model

Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, Jing Liu

arXiv:2306.09085v1cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Image-text pretraining is common, but it overlooks temporal context and video-text training data are limited in scale and quality. COSA concatenates image-text pairs into pseudo long-form video-paragraphs to model visual content and event-level cues. The authors report consistent downstream improvements and state-of-the-art results across multiple benchmarks.

  • Problem

    Image-text pretraining overlooks temporal contexts and correlations, while video-text training corpora are limited in scale and quality.

    Source: §1 · pp. 1–2

  • Method

    COSA sequentially concatenates multiple image-text pairs into pseudo long-form video-paragraphs with event-sentence correspondence for pretraining.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    COSA consistently improves downstream performance across video-language and image-text tasks and achieves new state-of-the-art results across benchmarks and model scales.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The reported findings support sample concatenation as an approach for learning vision-language correlations and temporal correspondence across image-text and video-language tasks.

    Source: §1 · p. 2, §5 · p. 10

    COSA has only been proven effective in the vision-language pretraining field.

    Source: §5 · p. 10

Abstract

from arXiv · show

Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations. To address this issue, we propose COSA, a COncatenated SAmple pretrained vision-language foundation model. COSA jointly models visual contents and event-level temporal cues using only image-text corpora. We achieve this by sequentially concatenating multiple image-text pairs as inputs for pretraining. This transformation effectively converts existing image-text corpora into a pseudo long-form video-paragraph corpus, enabling richer scene transformations and explicit event-description correspondence. Extensive experiments demonstrate that COSA consistently improves performance across a broad range of downstream tasks, including long-form/short-form video-text tasks and image-text tasks such as retrieval, captioning, and question answering. Notably, COSA achieves state-of-the-art results on various competitive benchmarks. Code and model are released at https://github.com/TXH-mercury/COSA.

1 Introduction

Image-text corpora dominate vision-language pretraining, but image-text models overlook temporal context needed for video-language reasoning. COSA concatenates image-text samples into pseudo long-form video-paragraphs to model visual content and event-level temporal correspondence, and reports improvements across downstream tasks.

1 Introduction

  • Image-text models overlook temporal contexts crucial for video-language reasoning, while video-text corpora are restricted in scale by costly collection and are far smaller than image-text corpora.The passage contrasts WebVid10M with LAION-2B and notes that image-text foundation models can nevertheless perform competitively on video tasks.Source: §1 · pp. 1–2
  • COSA dynamically concatenates randomly selected image-text samples in the same order for images and texts, creating pseudo long-form video-paragraphs with explicit event-sentence correspondence.The proposed corpus is described as offering richer scene transformations, less visual redundancy, and longer fine-grained captions than short-form video-text corpora.Source: §1 · p. 2
  • COSA is a unified foundation model intended to handle discriminative and generative tasks, including retrieval, captioning, and question answering.The model uses concatenated samples during pretraining.Source: §1 · p. 2
  • Ablations report consistent improvements across long- and short-form video-language and image-language tasks, and COSA achieves new state-of-the-art results across benchmarks and model scales.The authors attribute the gains to richer information in concatenated samples that supports vision-language correlations and temporal correspondence.Source: §1 · p. 2

2 Related Work

Prior video-language pretraining methods target temporal learning through specialized backbones, objectives, or data, while unified models commonly combine image and video corpora or adapt image-pretrained models. COSA instead uses image-text corpora alone and concatenates samples to create synthetic long-form training data.

Temporal Learning in Video-Language Pretraining (VidLP)

  • Earlier temporal-learning approaches used action-classification-pretrained backbones or objectives such as frame-order and masked-video modeling, while COSA concatenates samples to compose synthetic long-form data online.The authors identify limited action annotations, caption redundancy, and weak event-sentence correspondence as shortcomings of existing video-text corpora.Source: §2 · p. 3
  • Other approaches address these data and learning challenges by adding Kinetics-400, designing fine-grained alignment, or constructing long-form datasets from HD_VILA_100M.The passage names OmniVL, HiTeA, and LF_VILA, respectively.Source: §2 · p. 3

Unified Vision-Language Foundation Model

  • Earlier unified video-language models mixed image-text and video-text corpora, treating images as frozen videos or adding advanced backbones and specialized objectives.Frozen is cited as the first method to use a mixture of image-text and video-text corpora for video-language pretraining.Source: §2 · p. 3
  • Unified models commonly co-train on image and video corpora or train first on image-text data and then adapt for video, whereas COSA models both using image-text corpora only.The passage describes co-training and two-stage training as the two categories of large-scale unified models.Source: §2 · p. 3

3 Method

COSA combines a vision encoder and a flexible text encoder with online concatenation of samples into pseudo long-form inputs. Its training uses concatenated-sample objectives alongside single-sample objectives, with random sampling trading clip relevance for reduced redundancy.

3.1 Simple Model Architecture

  • COSA combines a ViT vision encoder with a BERT text encoder whose added cross-attention layers support cross-modal understanding or generation and are disabled for single-modality encoding.This design lets the text encoder act as a single-modality encoder, cross-modality encoder, and decoder.Source: §3.1 · p. 3

3.2 Concatenated Sample Training

  • For each image-text sample, COSA randomly selects three other batch samples by default and concatenates their images and texts in matching order to form a pseudo long-form input.Visual features receive learnable temporal embeddings and are concatenated sequentially; the text paragraph uses BERT positional embeddings to encode temporal relations.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
  • Random cross-sample concatenation creates independent clips with rapid scene changes, reducing redundancy but sacrificing clip relevance; enforcing relevance performed worse in the reported experiments.The same concatenation process can be applied to video-text corpora by sampling one frame from each video.Source: §3.2 · p. 4
  • COSA trains with concatenated-sample contrastive, matching, masked-language, and generation objectives, while retaining vanilla single-sample ITC and ITM objectives.CMLM masks 15% of paragraph tokens; CGM masks 60% and uses causal attention masks, while CITC aligns paired global representations and CITM classifies video-paragraph matches.Source: §3.2 · pp. 4–5
  • Table 1 presents COSA model configurations; its largest 1.2B model uses data bootstrapping and includes captions generated by an additionally trained captioner.CC14M combines CC3M, CC12M, COCO, SBU, and VG.Source: Table 1 · p. 5

4 Experiments

COSA is evaluated across video- and image-language tasks, with reported gains over comparison models and broad improvements over single-sample training. Ablations examine the effects of concatenation, training objectives, model settings, and sampling choices.

4.2 Comparison to State-of-the-Art

  • COSA-B (5M) outperforms similarly scaled models on all four text-to-video retrieval benchmarks, while large-scale COSA achieves the best R@1 on DiDeMo and ActivityNet.The prose reports comparable results on the other benchmarks against large-scale models.Source: §4.2 · pp. 6–7
  • COSA reports new state-of-the-art accuracy on four video QA benchmarks, new state-of-the-art video captioning on open- and domain-specific benchmarks, and new state-of-the-art image-text results on MSCOCO and Flickr30K.For video QA, the reported large-scale result uses 415M training data and 1.2B parameters; image captioning also reports the best MSCOCO SPICE score (27.0).Source: Table 3 · p. 6, Table 4 · p. 6, §4.2 · p. 7

4.3 Ablation Study

  • Across 12 video- and image-language benchmarks, COSA consistently improves over SST; removing cross-sample concatenation drops performance below the baseline, while preserving order further enhances several tasks.The reported benefits include long-form video retrieval, video captioning, and VIST.Source: Table 6 · p. 7, §4.3 · p. 8
  • CMLM and CGM improve all five benchmarks over SST, but replacing ITC and ITM with CITC and CITM degrades performance on every benchmark; default objectives perform best on most.Table 7 compares the different training objectives across retrieval, captioning, and QA tasks.Source: Table 7 · p. 8, §4.3 · p. 9
  • COSA consistently outperforms SST with Swin Transformer and WebVid2.5M settings; multiple-frame sampling helps except on VIST but is less effective than COSA.This robustness experiment changes the vision backbone and training corpus from CLIP-B and CC3M.Source: Table 8 · p. 8, §4.3 · p. 9
  • COSA’s performance gap over SST remains as training continues, and 30K-step COSA outperforms 120K-step SST on all benchmarks.The reported training times are 10 hours for COSA and 23 hours for SST.Source: Figure 3 · p. 8, §4.3 · p. 9
  • The selected concatenation number is nc = 3, which performs consistently well across benchmarks among tested values from 0 to 4.The tested range includes nc = 0, the SST baseline.Source: §4.3 · p. 9, Figure 4 · p. 9
  • Randomly grouped samples perform better than grouping the most similar samples, supporting reduced semantic relevance among concatenated samples.Alternatives select the most similar samples using image- or text-encoder features.Source: §4.3 · p. 9, Table 9 · p. 9

5 Conclusion and Limitation

COSA turns image-text corpora into pseudo long-form video-text corpora to promote event-level temporal learning. The authors report improvements across video- and image-language tasks, while noting that its effectiveness has only been demonstrated in vision-language pretraining.

5 Conclusion and Limitation

  • COSA transforms image-text corpora into pseudo long-form video-text corpora, enabling explicit event-sentence correspondence and promoting event-level temporal learning.The paper presents COSA as a concatenated-sample pretrained unified vision-language foundation model.Source: §5 · p. 9
  • Ablation studies report improvements on long-form and short-form video-language tasks and image-text tasks, with state-of-the-art results on multiple benchmarks.The authors hope the approach inspires research on co-learning multiple samples in other AI communities.Source: §5 · p. 10
  • COSA has only been proven effective in the vision-language pretraining field.Effectiveness beyond that field is not established in this conclusion.Source: §5 · p. 10

A Pretraining Settings

The pretraining settings for COSA models are presented in Table 10. For the largest model, captions in some corpora were replaced with descriptions generated by a separately trained captioner.

A Pretraining Settings

  • Table 10 presents the pretraining settings for all scales of COSA models.An asterisk marks corpora whose captions were replaced with descriptions generated by a separately trained captioner.Source: Table 10 · p. 10
  • For the largest COSA model, a separately trained captioner generated replacement captions for CC12M and LAION-102M using top-10 sampling.The captioner used COSA's architecture, was pretrained with generative modeling loss on CC4M, CC12M, and LAION-102M with original captions, then finetuned on MSRVTT, VATEX, MSVD, and MSCOCO.Source: A Pretraining Settings · p. 10

B Downstream Dataset Descriptions

The section describes downstream video, image, and question-answering benchmarks and their evaluation tasks and splits. It also points to the table of benchmark splits and to a separate table of COSA finetuning settings.

B Downstream Dataset Descriptions

  • COSA is evaluated on video datasets including MSRVTT, MSVD, LSMDC, DiDeMo, VATEX, YouCook2, ActivityNet Caption, TGIF, and TVC, and on MSCOCO, Flickr30K, and VQAv2.The section points to Table 11 for train, validation, and test splits.Source: B Downstream Dataset Descriptions · p. 10, Table 11 · p. 11
  • Video evaluations cover retrieval, captioning, and question answering, including paragraph-to-video retrieval on DiDeMo's temporally ordered annotations.The descriptions specify MSRVTT's 1K-A retrieval split, MSVD's official captioning split, and LSMDC's official retrieval split.Source: B Downstream Dataset Descriptions · p. 10
  • VATEX evaluations cover text-to-video retrieval and captioning, while YouCook2 is evaluated for captioning and ActivityNet Caption for retrieval and video question answering.The descriptions specify HGR's retrieval split for VATEX, official splits for YouCook2 and ActivityNet retrieval, and the Yu et al. split for ActivityNet QA.Source: B Downstream Dataset Descriptions · pp. 10–11
  • TGIF-Frame is used for video QA, TVC for vision-only video captioning, and MSCOCO and Flickr30K for image retrieval; MSCOCO also supports captioning and VQAv2 supports question answering.The descriptions specify official or Karpathy splits for these evaluations and train-plus-validation data for VQAv2 training.Source: B Downstream Dataset Descriptions · p. 11

C Finetuning Settings

  • Specific COSA finetuning settings, including learning rate, batch size, training epochs, and sampled frames, are presented in Table 12.The passage identifies the settings but does not provide their values.Source: C Finetuning Settings · p. 11

D.1 Text-to-Video Retrieval on VATEX Benchmark

COSA-B with a Swin-B vision backbone outperformed CLIP4Clip with a CLIP-ViT-B backbone. COSA also surpassed VALOR despite using less training data.

D.1 Text-to-Video Retrieval on VATEX Benchmark

  • COSA-B with Swin-B outperformed CLIP4Clip with CLIP-ViT-B, and COSA surpassed VALOR despite using less training data.The authors say these comparisons further demonstrate the effectiveness of their method.Source: §D.1 · p. 11

D.2 Qualitative Comparison on VIST Benchmark

On VIST, COSA generates comprehensive stories across zero-shot and finetuning settings, whereas SST produces limited or duplicated captions depending on the setting.

D.2 Qualitative Comparison on VIST Benchmark

  • COSA generates comprehensive stories for multiple frames in both zero-shot and finetuning settings on VIST.Figure 5 compares COSA with SST on the VIST benchmark.Source: §D.2 · pp. 11–12, Figure 5 · p. 13
  • Under zero-shot evaluation, SST produces only one sentence describing limited content, unlike COSA’s multi-frame stories.The comparison is qualitative, as shown in Figure 5.Source: §D.2 · pp. 11–12, Figure 5 · p. 13
  • Under finetuning, SST is prone to duplicated captions, while COSA generates comprehensive stories.Figure 5 presents the qualitative comparison on VIST.Source: §D.2 · pp. 11–12, Figure 5 · p. 13
Loading 2306.09085v1…