PaperlayerView exact revision on arXiv

Source-linked AI summary

Align and Attend: Multimodal Summarization with Dual Contrastive Losses

Bo He, Jun Wang, Jielin Qiu, Trung Bui, Abhinav Shrivastava, Zhaowen Wang

arXiv:2303.07284v3cs.CV

TL;DR

Existing multimodal summarization methods can overlook temporal correspondence between modalities and intrinsic cross-sample correlations. A2Summ aligns and fuses multimodal inputs and applies two contrastive losses; it achieves the best performance on SumMe and TVSum overall, with a stated TVSum F1 exception versus CLIP-It.

  • Problem

    Existing multimodal summarization methods do not exploit temporal correspondence between modalities, and prior correlation strategies require large amounts of annotated multimodal data.

    Source: §1 · p. 1

  • Method

    A2Summ uses alignment-guided self-attention to fuse temporally corresponding modalities and inter-sample and intra-sample contrastive losses to model cross-modal correlations.

    Source: §3.3 · p. 4

  • Results

    A2Summ achieves the best performance on both SumMe and TVSum; on TVSum, it is slightly worse than CLIP-It on F1 but higher on the other two metrics.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    BLiSS adds livestream videos and transcripts with annotated frame and sentence summaries for further multimodal summarization research.

    Source: §1 · p. 2, §5 · p. 8

    A2Summ’s Transformer self-attention has quadratic computation cost with respect to input sequence length.

    Source: D. Limitation and Future Work · p. 13

Abstract

from arXiv · show

The goal of multimodal summarization is to extract the most important information from different modalities to form output summaries. Unlike the unimodal summarization, the multimodal summarization task explicitly leverages cross-modal information to help generate more reliable and high-quality summaries. However, existing methods fail to leverage the temporal correspondence between different modalities and ignore the intrinsic correlation between different samples. To address this issue, we introduce Align and Attend Multimodal Summarization (A2Summ), a unified multimodal transformer-based model which can effectively align and attend the multimodal input. In addition, we propose two novel contrastive losses to model both inter-sample and intra-sample correlations. Extensive experiments on two standard video summarization datasets (TVSum and SumMe) and two multimodal summarization datasets (Daily Mail and CNN) demonstrate the superiority of A2Summ, achieving state-of-the-art performances on all datasets. Moreover, we collected a large-scale multimodal summarization dataset BLiSS, which contains livestream videos and transcribed texts with annotated summaries. Our code and dataset are publicly available at ~\url{https://boheumd.github.io/A2Summ/}.

1. Introduction

Existing multimodal summarization methods often miss temporal correspondence and provide limited modeling of cross-modal correlations. A2Summ combines alignment-guided fusion and dual contrastive losses, alongside BLiSS, a livestream dataset with video and transcript summaries.

1. Introduction

  • Multimodal summaries can provide useful information for redundant long videos, including livestreams and product-review videos.The introduction also names news and vlogs as potential application contexts.Source: §1 · p. 1
  • Existing multimodal summarization methods either produce a single-modality summary or fail to exploit temporal correspondence and intrinsic cross-modal correlations.The authors also note that prior correlation strategies require large amounts of annotated multimodal data, which are difficult to obtain.Source: §1 · p. 1
  • A2Summ uses alignment-guided self-attention to fuse temporally corresponding modalities and inter-sample and intra-sample contrastive losses to model cross-modal correlations at different granularities.The inter-sample loss contrasts matched and unmatched video-text pairs; the intra-sample loss uses similarities between ground-truth summaries and hard negatives.Source: §1 · p. 2
  • BLiSS contains livestream videos and transcripts annotated with summaries by selecting important frames and sentences.The authors motivate it in part by livestreams’ longer duration and slower content changes compared with existing short-video datasets.Source: §1 · p. 2

2. Related Work

Related work spans unimodal video and text summarization, as well as multimodal methods that often use extra modalities but output summaries from only one. Figure 2 depicts A2Summ’s multimodal inputs, alignment and fusion, and inter-sample contrastive loss.

2. Related Work

  • Video summarization includes unsupervised methods using hand-crafted features or reward functions and supervised approaches such as temporal-interest detection and joint moment localization.The cited video methods also include LSTM/GCN modeling of frame- and shot-level dependencies.Source: §2 · p. 2
  • Text summarization includes extractive methods that select salient input parts and abstractive methods that paraphrase important content; these approaches are designed for unimodal input.StreamHover is described as an unsupervised transcript-summarization model for livestreams.Source: §2 · p. 2
  • Many multimodal methods use complementary modalities to improve primary-modality representations but generate summaries in only one modality.Prior multimodal-output work includes models for text-image and text-video summaries, as well as datasets with audio and transcripts.Source: §2 · p. 2
  • Figure 2 shows video frames and text sequences as inputs, alignment-guided self-attention for fusion, and an inter-sample loss that distinguishes matched from unmatched pairs.The caption identifies B as the batch size.Source: Figure 2 · p. 3

3. Method

A2Summ processes video frames and transcribed sentences together, using their timestamps to align cross-modal attention. It combines modality-specific scoring with contrastive objectives across samples and within each sample.

3.1. Overview

  • A2Summ takes untrimmed video frames and transcribed sentences as input, using sentence start and end times to align the modalities.It extracts and projects frame and sentence features into a shared embedding space, then adds position and segment embeddings.Source: §3.1 · p. 3, §3.2 · p. 3

3.3. Multimodal Alignment and Fusion

  • Video and text features attend globally within each modality, while cross-modal attention is restricted to frames and sentences in the same time segment.The alignment-guided mask is applied to the self-attention matrix; modality-specific experts and score-prediction branches follow the attention blocks.Source: §3.3 · p. 4

3.4. Loss Function

  • Training combines classification losses for video and text with inter-sample video-text matching across a batch.The inter-sample objective increases similarity for real video-text pairs and decreases it for incorrect cross-batch pairs.Source: §3.4 · p. 4
  • Within each sample, the intra-sample loss contrasts key frames and sentences against hard negatives selected from high-scoring timesteps outside expanded key-frame labels.Figure 3 illustrates the selection procedure and the loss between selected video and text pairs; adjacent frames are excluded to avoid treating similar content as negatives.Source: §3.4 · p. 5, Figure 3 · p. 5

4. Experiments

Experiments compare A2Summ across standard video, multimodal news, and BLiSS livestream datasets, then examine component contributions and example summaries.

4.1. Datasets and Implementation Details

  • Experiments use SumMe, TVSum, Daily Mail, CNN, and BLiSS, with F1, rank correlations, ROUGE, and cosine similarity evaluated for the corresponding tasks.The standard video datasets use pre-extracted GoogLeNet features and generated frame captions; BLiSS uses CLIP and RoBERTa features.Source: §4.1 · p. 6

4.2. BLiSS Dataset

  • BLiSS contains 628 livestream videos, divided into five-minute clips for annotation, and totals 13,303 samples and 1,109 video hours.Annotators selected key transcript sentences and wrote clip summaries; key frames were selected by matching frames to website thumbnail animations.Source: §4.2 · p. 6

4.3. Results

  • A2Summ achieves the best performance on SumMe and TVSum, except TVSum F1 where it is slightly below CLIP-It; it also beats iPTNet across metrics without extra data.The comparison includes F1, Kendall’s τ, and Spearman’s ρ.Source: §4.3 · pp. 6–7, Table 2 · p. 7
  • On CNN and Daily Mail, A2Summ beats M2SM by 3% and 2.4% in ROUGE-1 F1, respectively, and video summarization methods by 1%.Because these datasets lack time information, the model uses the two contrastive losses without alignment-guided self-attention.Source: §4.3 · p. 7, Table 1 · p. 7
  • On BLiSS, A2Summ achieves the best video cosine similarity and exceeds SummaRuNNer and BART by at least 3% on all ROUGE scores.The comparison includes both extractive and abstractive text summarization methods.Source: §4.3 · p. 7, Table 3 · p. 8

4.4. Ablation Studies

  • On SumMe, adding alignment improves F1 by 1%, intra-sample contrastive loss raises performance by 2.5%, and all three components raise it from 50.5% to 55.0%.The authors report consistent gains from either contrastive loss and describe the two losses as complementary.Source: §4.4 · p. 8, Table 4 · p. 8

4.5. Visualization

  • In TVSum examples, alignment and dual contrastive losses address baseline errors including missed important segments and inaccurate summary boundaries.The figure’s gray histogram shows ground-truth frame-importance scores; baseline omits alignment and both contrastive losses.Source: §4.5 · p. 8, Figure 4 · p. 8

5. Conclusion

A2Summ is a unified transformer framework that aligns modalities using temporal correspondences and models inter- and intra-sample cross-modal information with dual contrastive losses. Experiments validate the framework, and the paper introduces a livestream video-and-transcript dataset.

5. Conclusion

  • A2Summ aligns modalities using time correspondences and introduces dual contrastive losses for inter-sample and intra-sample cross-modal information.The framework is a unified transformer-based model.Source: §5 · p. 8
  • Experiments on multiple datasets validate the effectiveness of A2Summ.Source: §5 · p. 8
  • The authors collected a large-scale multimodal summarization dataset focused on livestream videos and transcripts.They hope it will benefit further research.Source: §5 · p. 8

Appendix

The appendix points readers to additional dataset-collection details, implementation settings, qualitative results, and a discussion of limitations and future work.

Appendix

  • The appendix covers BLiSS collection details, dataset-specific implementation settings, qualitative results, and limitations and future work.Source: Appendix · p. 13

A. BLiSS Dataset

BLiSS contains livestream videos with English transcripts and metadata, organized into aligned five-minute clips and annotated for text and video summaries. The dataset is larger in video duration than the compared datasets and has more text tokens than Daily Mail and StreamHover.

A. BLiSS Dataset

  • BLiSS contains 628 livestream videos with English transcripts and creator-provided metadata; audio is available but unused because transcripts capture most audio information.Source: A. BLiSS Dataset · p. 13
  • Transcripts were sentence-segmented and timestamped, then videos were divided into five-minute clips with aligned frames and transcript sentences.Very short-duration transcripts likely caused by broken words or speech-recognition failures were removed.Source: A. BLiSS Dataset · p. 13
  • Annotators selected about 5 to 10 keywords per clip; keyword-containing sentences serve as extractive summaries, and annotators also wrote abstractive summaries.Source: A. BLiSS Dataset · p. 13
  • Keyframes were selected by matching each thumbnail-animation frame to its most similar frame in the video.Source: A. BLiSS Dataset · p. 13
  • BLiSS contains 1,109 hours of total video duration and 5.5M text tokens, with token count described as much larger than Daily Mail and StreamHover.Figure 5 shows uniformly sampled video frames, annotated keyframes, sentence-level transcripts, an abstractive summary, and extractive key sentences with ground-truth keywords marked blue.Source: A. BLiSS Dataset · p. 13

B. Experiment Details

The appendix lists separate training settings for multimodal, video-summarization, and BLiSS datasets, with a shared contrastive-pair expansion size across datasets.

B. Experiment Details

  • Daily Mail and CNN used batch size 4, learning rate 2e-4, 100 epochs, L=2, and r=8; β/λ were 0.001/0.001 for Daily Mail and 0/0 for CNN.The listed weight-decay values were 1e-7 and 1e-5.Source: B. Experiment Details · p. 13
  • SumMe and TVSum used batch size 4, learning rate 1e-3, 300 epochs, L=2, and r=16; β=0.1, while λ was 3 for SumMe and 1 for TVSum.The listed weight-decay values were 1e-3 and 1e-5.Source: B. Experiment Details · p. 13
  • BLiSS training used batch size 64, learning rate 1e-3, weight decay 1e-7, 50 epochs, L=6, r=4, β=0.01, and λ=0.001.Source: B. Experiment Details · p. 13
  • The expansion size on both sides of keyframes and key sentences in contrastive-pair selection was 4 across all datasets.Source: B. Experiment Details · p. 13

C. More Qualitative Results

On BLiSS examples, A2Summ predicts key sentences more accurately and faithfully than the baseline, while its predicted keyframes qualitatively capture important scenes.

C. More Qualitative Results

  • A2Summ predicts BLiSS key sentences more accurately and faithfully than the baseline, while its keyframes qualitatively capture important scenes.The videos change slowly, so their frames often share similar visual content.Source: C. More Qualitative Results · p. 13

D. Limitation and Future Work

A2Summ’s self-attention has quadratic computation cost as input sequences grow, and producing video and text annotations is laborious. The authors suggest unsupervised or self-supervised multimodal summarization as future work.

D. Limitation and Future Work

Loading 2303.07284v3…