Source-linked AI summary
How2: A Large-scale Dataset for Multimodal Language Understanding
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, Florian Metze
TL;DR
How2 addresses the limited availability of pervasive multimodal resources by providing instructional videos paired with speech, subtitles, translations, summaries, and visual context. The paper constructs the dataset, releases sequence-to-sequence baselines and code for several tasks, and finds that action-level visual features improve automatic speech recognition, video summarization, and speech-to-text translation. How2 is presented as a resource for broader multimodal language research.
Problem
Existing multimodal datasets are typically single-task, limiting resources for developing general-purpose multimodal language understanding.
Method
The paper introduces How2 and releases sequence-to-sequence baselines and code for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization.
Results
Action-level visual features improve automatic speech recognition, video summarization, and speech-to-text translation.
Takeaways & Limitations
How2 provides data and code intended to stimulate research on multimodal natural language tasks and deepen understanding of multimodality in language processing.
Abstract
from arXiv · showhide
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges, to obtain a deeper understanding of multimodality in language processing.
1 Introduction
How2 addresses the lack of pervasive multimodal resources by combining instructional videos with aligned speech, subtitles, translations, summaries, and visual context. Baseline experiments show that action-level visual features improve several multimodal language tasks and help resolve textual ambiguities.
- Motivation: Existing multimodal datasets typically target single tasks, limiting progress toward general-purpose multimodal language understanding.How2 is designed as a more pervasive multimodal resource spanning multiple task types.
- How2 dataset: How2 pairs instructional videos and spoken utterances with English subtitles, crowdsourced Portuguese translations, and English video summaries.The dataset is naturally occurring and integrates multiple modalities at utterance and video levels.
- Multimodal disambiguation: Visual and audio context can disambiguate subtitle meanings that text alone leaves unresolved, such as whether “green” refers to a color or a golf surface.The example uses visual evidence of grass and a flag pole, or audio of an outdoor golf shot.
- Baseline findings: Additional modalities provide useful cues for ASR by filtering activity-related noise, representing environmental characteristics, and adapting language models to domains.The passage identifies object, motion, scene, and entity-level visual information as relevant cues.
- Baseline findings: Action-level visual features improve automatic speech recognition, video summarization, and speech-to-text translation in baseline experiments.The authors present these results as evidence of How2’s potential for future multimodal research.
2 How2 Dataset
How2 is a large, naturally occurring instructional-video dataset aligned with English speech, subtitles, Portuguese translations, and visual content. Its construction combines sentence-level audiovisual alignment with crowdsourced translation, producing a resource whose annotations and benchmarks support multimodal research.
- Dataset scale: How2 contains 79,114 instructional videos totaling 2,000 hours, with an average duration of 90 seconds and English subtitles.The corpus can be recreated from released scripts and metadata.
- Data construction: English subtitles were re-segmented into full sentences, aligned to speech at the word level, and matched with visual features from corresponding video clips.Segment durations are summarized in Figure 2b.
- Translation annotation: The 300h subset received Portuguese post-edited translations from 200 workers, with no worker reaching the 5,000-segment limit.The corpus used Figure Eight crowdsourcing, with workers restricted to Portugal or Brazil.
- Translation annotation: 18% of the 300h subset was discarded and re-annotated, at a total data-collection cost of US$ 8,771.The reported cost covers collection completed at the time of writing.
- Validation: Training English–Portuguese neural MT on 300h of machine-generated data reduced performance by about 1 BLEU point versus post-edited translations.Evaluation used expert-validated post-edited translations, supporting the use of post-edited data.
- Corpus analysis: The dataset uses 22 LDA-based topics to represent the 300h subset, whose topic and segment-duration characteristics are similar to the full 2000h corpus.Topics were hand-labeled from top words in each cluster.
3 Experiments
The paper evaluates How2 with sequence-to-sequence baselines across speech recognition, translation, speech-to-text translation, and summarization. Multimodal variants incorporate visual information through adaptive training or hierarchical attention, with action-level features used for multimodal summarization.
- Automatic speech recognition: Multimodal ASR adds a learned video-specific bias to speech features through a jointly trained linear adaptation layer.The adaptation layer increases model size by less than 1%.
- Machine translation: Multimodal MT applies the adaptive scheme to word embeddings rather than speech features.This adaptation layer increases model size by 8%.
- Speech-to-text translation: Speech-to-text translation directly maps English speech to Portuguese with the ASR architecture and a different target vocabulary, while its multimodal variant uses the same adaptive scheme.The task avoids an intermediate English transcription in the direct translation setup.
- Summarization: Multimodal summarization combines textual and visual modalities with hierarchical attention over a sequence of action-level features.This design increases model size by 14%, unlike the average-pooled visual features used in other experiments.
4 Related work
Prior multimodal datasets largely center on image or video captioning and often have limited linguistic diversity or nonparallel multilingual annotations. How2 addresses these gaps with larger bilingual instructional-video data and supports multimodal ASR, speech-to-text translation, and summarization.
- Image and translation datasets: Image-captioning and multimodal-translation datasets such as Flickr8K, Flickr30K, MSCOCO, IAPR-TC12, and Multi30K often lack syntactic and semantic diversity.Several captioning datasets were extended with translations into additional languages.
- Video description: Existing video-description resources include MSR-VTT, LSMDC, and MSVD, but MSVD’s non-English descriptions are not parallel and two-thirds remain in English.How2 provides a larger amount of data in two languages.
- Multimodal speech tasks: How2 enables multimodal ASR using images as acoustic and linguistic context, extending beyond lipreading’s non-semantic fusion of modalities.The paper identifies How2 as the first dataset supporting this multimodal ASR setting.
- Multimodal speech tasks: How2 is described as the only currently available corpus for multimodal speech-to-text translation.Prior speech-to-text translation commonly uses Fisher-Callhome and Audiobooks.
- Summarization: Multimodal abstractive summarization lacks well-established benchmarking datasets, with prior resources including a 500-video English news corpus and image-text collections.Traditional text summarization commonly relies on CNN/Daily Mail, Gigaword, and Document Understanding Conference data.
5 Conclusions
The paper introduces How2 as a multimodal instructional-video collection with English subtitles and crowdsourced Portuguese translations, and releases sequence-to-sequence baselines across several multimodal language tasks. The accompanying data and code are intended to support further research on multimodal language processing.
- Contributions: How2 combines instructional videos, English subtitles, crowdsourced Portuguese translations, and sequence-to-sequence baselines for multiple multimodal language tasks.The baselines cover machine translation, automatic speech recognition, spoken language translation, and multimodal summarization.
- Implications: The released data and code are intended to stimulate research on multimodal natural language challenges and deepen understanding of multimodality in language processing.
A.1 How2 Examples
How2 examples pair video-frame content with utterances and summaries, showing how visual context corresponds to spoken instructional content.
- A.1 How2 Examples: How2 examples demonstrate correspondence among video frames, utterances, and video summaries.The examples include hairdressing and cooking scenes whose visual elements provide contextual information.
- A.1 How2 Examples: Visual context can enrich instructional utterances through objects, people, and surrounding scenes.The hair product, hairdresser, salon, cooking devices, and kitchen are identified as useful contextual elements.
- A.1 How2 Examples: The dataset includes varied instructional situations, including hairdressing and cooking demonstrations.
A.2 Modality Alignment and Data Checks
How2 aligns text, audio, and video at token and utterance levels, then validates alignment quality and clarifies that its transcriptions are canonical rather than verbatim.
- A.2 Modality Alignment and Data Checks: Token-level timestamps link text, audio, and video, while utterance-level start and end times support synchronized segmentation.
- A.2 Modality Alignment and Data Checks: Viterbi alignment with Kaldi’s WSJ GMM/HMM acoustic model estimates sentence boundaries in the audio.These alignments are used to segment both audio and video according to utterances.
- A.2 Modality Alignment and Data Checks: Manual inspection found that randomly chosen utterances with good alignment scores were accurately aligned in the selected 300 h subset.The WSJ model performed best relative to the other evaluated models.
- A.2 Modality Alignment and Data Checks: How2 transcriptions are canonical forms derived from subtitles rather than verbatim records of spoken content.This is suitable for the proposed uses but may slightly increase overall word error rates in speech-to-text tasks.
A.3 Feature Extraction and Processing
How2 represents speech with normalized acoustic features, video with action-recognition features, and text with separately trained English and Portuguese tokenizations.
- A.3 Feature Extraction and Processing: Speech features combine 40-dimensional filter banks with 3-dimensional pitch features into 43-dimensional vectors.Features are extracted from 16 kHz speech using 25 ms windows and 10 ms frame shifts, followed by per-video CMVN.
- A.3 Feature Extraction and Processing: Visual processing extracts a 2048-dimensional vector every 16 frames using a CNN trained to recognize 400 actions.For sentence-level experiments, these sequences are average pooled into one vector representing the video.
- A.3 Feature Extraction and Processing: Text is normalized, lowercased, and stripped of punctuation before tokenization.Separate English and Portuguese SentencePiece models produce 5K-word vocabularies, while summarization uses word-level tokens.
A.4 Architecture Details
The baselines use sequence-to-sequence architectures across speech, translation, and summarization, with multimodal variants adapting speech or text representations and preserving action-level visual sequences for summarization.
- A.4 Architecture Details: The ASR baseline uses a 6-layer pyramidal encoder with subsampling rate 4 and a 2-layer conditional GRU decoder.The decoder uses feed-forward attention, tied embeddings, and encoder-state initialization through a tanh layer.
- A.4 Architecture Details: Multimodal ASR applies a learned linear transformation of video features as a visual bias added to speech features at each time step.
- A.4 Architecture Details: Multimodal ASR initializes from a trained ASR model and jointly optimizes all parameters, including the adaptation parameters.
- A.4 Architecture Details: The machine-translation baseline is a sequence-to-sequence model with a 2-layer bidirectional GRU encoder and conditional GRU decoder.Dropout with probability 0.3 is applied at the specified embedding, encoding, and softmax locations.
- A.4 Architecture Details: Multimodal summarization combines textual and visual modalities with hierarchical attention.Unlike the other multimodal architectures, it uses action-level visual feature sequences rather than average-pooled vectors.
- A.4 Architecture Details: Training uses ADAM, gradient clipping, validation-based early stopping, and task-specific WER, BLEU, and ROUGE-L evaluation.The learning rate is halved when validation performance does not improve.