Source-linked AI summary
LRS3-TED: a large-scale dataset for visual speech recognition
Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman
TL;DR
Visual speech recognition lacks a large-scale common benchmark, and existing datasets have usage restrictions. This paper introduces LRS3-TED, a multi-modal corpus collected through an automated pipeline from TED and TEDx videos with aligned visual, audio, and text data. The corpus is intended for lip reading, audio-visual speech recognition, speech enhancement, and related learning tasks.
Problem
Existing lip-reading datasets have restrictions, making it difficult to compare systems because no large-scale common benchmark dataset is available.
Method
The paper constructs LRS3-TED with a multi-stage pipeline that collects TED and TEDx face tracks, subtitles, audio, and word-level alignments.
Results
LRS3-TED contains over 400 hours of video from 5,594 TED and TEDx talks, with corresponding face tracks, subtitles, and word alignment boundaries.
Takeaways & Limitations
The corpus is useful for lip reading, audio-visual speech recognition, video-driven speech enhancement, and other audio-visual learning tasks.
Takeaways & Limitations
Train-val and test are divided by videos, but speaker identities are not explicitly labeled and some identities may overlap across splits.
Abstract
from arXiv · showhide
This paper introduces a new multi-modal dataset for visual and audio-visual speech recognition. It includes face tracks from over 400 hours of TED and TEDx videos, along with the corresponding subtitles and word alignment boundaries. The new dataset is substantially larger in scale compared to other public datasets that are available for general research.
1. Introduction
Visual speech recognition is challenging, while progress has been enabled by deep neural networks and larger datasets. LRS3-TED is released as a larger, common benchmark intended to improve comparison across lip-reading systems.
- Visual speech recognition is difficult for humans and automated systems.
- Deep neural networks and large-scale datasets have recently improved automated lip-reading performance.
- Restrictions on datasets such as LRW and LRS2-BBC limit industrial research use and hinder system comparisons.
- LRS3-TED is intended to provide a common benchmark dataset larger than available alternatives.
2. LRS3-TED dataset
LRS3-TED is a large audio-visual speech dataset built from TED and TEDx videos, with face tracks, subtitles, and word-level alignments. Its pipeline selects suitable talks, detects and tracks faces, aligns audio and text, and separates data by videos.
- Over 400 hours of video were extracted from 5,594 English TED and TEDx talks downloaded from YouTube.
- The dataset provides 224×224 face-track videos at 25 fps, 16-bit 16kHz mono audio, transcripts, and word alignment boundaries.
- The dataset is divided into pre-train, train-val, and test sets, with the test set completely independent and train-val and test divided by videos.
- The multi-stage pipeline detects faces with an SSD-based CNN, forms tracks within shots, aligns subtitles to audio with P2FA, and double-checks alignments using Kaldi-based ASR.
- TED and TEDx talks were selected for speaker diversity, relatively infrequent shot changes, and longer uninterrupted face tracks.
- The pre-train set uses full-duration face-track videos and subtitles from the train-val source videos, with word-level alignment boundaries.
3. Conclusion
The paper presents the LRS3-TED audio-visual corpus as useful for lip reading, audio-visual speech recognition, speech enhancement, and other audio-visual learning tasks. It also points to reported lip-reading evaluations on the dataset.
- LRS3-TED supports lip reading, audio-visual speech recognition, video-driven speech enhancement, and other audio-visual learning tasks.
- Reported evaluations assess recent lip-reading models on the dataset.