Multimodal Abstractive Summarization for How2 Videos

• Video summaries can provide useful discovery context when associated text is absent or does not capture subtle differences between related videos.Source: §1 · p. 1
• Multimodal summarization combines textual and visual modalities, while prior multimodal video-summary data described here were small-scale news videos without human-annotated audio transcripts.Source: §1 · p. 1
• The work studies multimodal summaries of open-domain instructional videos using How2, which provides human-annotated summaries across varied topics.Source: §1 · p. 2
• Text, visual, and audio-transcript information are considered as sources for generating natural-language descriptions of video content.Source: §1 · p. 2
• The paper introduces Content F1 as an evaluation metric intended to suit multimodal abstractive summarization.Source: §1 · p. 2

• How2 contains about 2,000 hours of instructional videos across domains, each accompanied by a transcript and a viewer-oriented summary of two to three sentences.Source: §2 · p. 2
• Text and vision provide complementary information: a summary can identify a Cuban breakfast recipe even though that is not stated in the transcript.Source: §2 · p. 2
• The corpus has 73,993 training, 2,965 validation, and 2,156 test videos; transcripts average 291 words and summaries 33 words.Source: §2 · p. 2
• Video inputs are sequential 2,048-dimensional features extracted every 16 non-overlapping frames using a pretrained ResNeXt-101 action-recognition model.Source: §2 · pp. 2–3
• Speech-recognition transcripts from ASpIRE and EESEN are used as text input, with 35.4% word error rate on How2 test data, mostly attributed to normalization issues.Source: §2 · p. 3

• The RNN sequence-to-sequence model encodes text or video features with attention and generates summaries through a decoder RNN.Source: §3 · p. 3
• The pointer-generator model is included as an abstractive summarization alternative alongside the sequence-to-sequence model.Source: §3 · p. 3
• Hierarchical attention first computes separate context vectors for text and video, then treats them as states of another encoder to compute a combined vector.Source: §3 · p. 3
• For video represented by sequential action features rather than one averaged vector, an RNN layer helps capture context.Source: §3 · p. 3

• ROUGE-L measures the longest common sequence between reference and generated summaries, while Content F1 is also used to evaluate the outputs.Source: §4 · p. 3
• Content F1 computes F1 over aligned content words after removing function words and task-specific stop words from reference and hypothesis summaries.Source: §4 · pp. 3–4
• Table 1 covers ROUGE-L and Content F1 for random, extracted, nearest-neighbor, text-only, pointer-generator, ASR-transcript, video-only, and text-and-video models.Source: Table 1 · p. 4
• Table 2 reports human-evaluation scores for informativeness, relevance, coherence, and fluency.Source: Table 2 · p. 4
• Human annotators rate informativeness, relevance, coherence, and fluency on a 1–5 scale for randomly sampled 500 test videos.Source: §4 · p. 4

5 Experiments and Results

• A fluent randomly sampled language-model baseline receives high ROUGE despite unrelated content, whereas the nearest-neighbor baseline has Content F1 similar to the rule-based baseline.The authors present the latter similarity as evidence of related content and of Content F1's utility.Source: §5 · p. 4
• The text-only model performs best with the complete 650-token transcript, while pointer-generator networks do not outperform sequence-to-sequence models on this data.The authors suggest the abstractive summaries and limited input-output n-gram overlap may explain the pointer-generator result.Source: §5 · p. 4
• The automatic-transcription model performs competitively with video-only models but degrades noticeably relative to ground-truth-transcription summarization.The authors attribute the degradation to substantial ASR errors in distant-microphone, open-domain speech recognition.Source: §5 · pp. 4–5
• The hierarchical-attention model combining text and video achieves the highest scores, and multimodal models with hierarchical attention lead on all three human evaluation measures.The paper also reports that action-only models reach almost competitive ROUGE and Content F1 scores against the text-only model.Source: §5 · p. 5
• Most model summaries are shorter than human annotations, with the action-only model shortest; ground-truth-text and ASR-text systems have similar length distributions.The authors note that the ROUGE-L and Content F1 improvements for these text-feature systems stem from differences in content rather than length.Source: §5 · p. 5

6 Conclusions

• The paper presents baseline models for abstractive summaries of open-domain How2 videos, including a video-only model that performs competitively with a text-only model.The authors identify multi-document, multi-video summaries and end-to-end models from video audio, rather than pretrained ASR text, as future directions.Source: §6 · p. 5
• Content F1 evaluates video summaries designed as teasers or highlights for viewers rather than condensed versions of their input.The paper defines and demonstrates the metric's quality for this evaluation purpose.Source: §6 · p. 5

A.1 Experimental Setup

• The text encoder has two bidirectional 256-unit GRU layers, and the decoder has two Conditional GRU layers.The models are optimized with Adam at learning rate 4 · 10−4; the rate is halved when validation performance fails to improve, for up to 50 epochs.Source: §A.1 · p. 8
• Inputs are limited to 600 tokens, except for the best text-only model, and the vocabulary contains the 20,000 most frequent words.The authors report that this vocabulary outperformed subword-based vocabularies in their experiments.Source: §A.1 · p. 8
• The setup reports frequent words in transcripts and summaries in Table A1.The table concerns input and output vocabulary, rather than model configuration.Source: Table A1 · p. 8

A.2 Frequent Words in Transcripts and Summaries

• Transcript words reflect conversational, spontaneous speech; summary words reflect their descriptive nature.Table A1 lists the most frequently occurring words in each.Source: §A.2 · p. 8

A.3 Output Examples from Different Models

• Text-only output is fluent and close to the reference, while the action-feature RNN adds relevant equipment detail; the random baseline is fluent but incorrect.The action-only variant without an RNN is relevant but less detailed, and the nearest-neighbor example is about knot tying rather than fishing.Source: §A.3 · p. 8, Table A2 · p. 9

A.4 Attention Analysis

• Attention is lower during introductions and paper-only close-ups, higher over brush strokes, and absent on black frames; the model also uses camera-angle cuts effectively.Source: §A.4 · pp. 8–10

A.5 Human Evaluation Details

• Human evaluation compared unimodal and multimodal summaries with references on informativeness, relevance, coherence, and fluency, using three workers per example.The 129 annotators were recruited from English-speaking countries and were not told which modality produced each candidate.Source: §A.5 · p. 10