Source-linked AI summary
LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
Shuang Yang, Yuanhang Zhang, Dalu Feng, Mingmin Yang, Chenhao Wang, Jingyun Xiao, Keyu Long, Shiguang Shan, Xilin Chen
TL;DR
Lip-reading lacks large-scale Mandarin benchmarks that reflect natural speech and imaging variability. The paper introduces LRW-1000, constructs it from diverse television data, and evaluates representative methods, finding persistent challenges across scale, pose, and word duration.
Problem
Few large-scale lip-reading datasets existed, and Mandarin lip-reading datasets were particularly scarce.
Method
The paper constructs LRW-1000 from naturally distributed television data and evaluates representative lip-reading methods across multiple settings.
Results
LRW-1000 contains 1,000 classes, 718,018 samples, and more than 2,000 speakers; analyses show challenges involving image scale, pose, and word duration.
Takeaways & Limitations
LRW-1000 provides a large-scale, naturally distributed Mandarin benchmark for studying lip-reading under practical variability.
Abstract
from arXiv · showhide
Large-scale datasets have successively proven their fundamental importance in several research fields, especially for early progress in some emerging topics. In this paper, we focus on the problem of visual speech recognition, also known as lipreading, which has received increasing interest in recent years. We present a naturally-distributed large-scale benchmark for lip reading in the wild, named LRW-1000, which contains 1,000 classes with 718,018 samples from more than 2,000 individual speakers. Each class corresponds to the syllables of a Mandarin word composed of one or several Chinese characters. To the best of our knowledge, it is currently the largest word-level lipreading dataset and also the only public large-scale Mandarin lipreading dataset. This dataset aims at covering a "natural" variability over different speech modes and imaging conditions to incorporate challenges encountered in practical applications. It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up. Besides providing a detailed description of the dataset and its collection pipeline, we evaluate several typical popular lipreading methods and perform a thorough analysis of the results from several aspects. The results demonstrate the consistency and challenges of our dataset, which may open up some new promising directions for future work.
I. INTRODUCTION
Lip-reading recognizes speech content from video alone, but progress has been hindered by few large-scale datasets and the difficulty of natural conditions. LRW-1000 addresses this gap with a large, naturally distributed Mandarin benchmark and analyses of current methods.
- Motivation: Large-scale datasets were previously scarce, creating a major obstacle to progress in lip-reading.
- Dataset contribution: 718,018 samples across 1,000 classes and more than 2,000 speakers make LRW-1000 a large-scale Mandarin word-level lip-reading benchmark.Classes correspond to syllables of Mandarin words, with over 1 million Chinese character instances covering 286 syllables.
- Dataset contribution: LRW-1000 is presented as the largest word-level lip-reading dataset and the only large-scale Mandarin lip-reading dataset.
- Naturally distributed data: Natural variability spans lighting, resolution, pose, speech rate, age, gender, make-up, class frequency, and sample duration.These properties are intended to reflect practical settings rather than fixed class sizes and word lengths.
- Evaluation: The benchmark evaluates three popular model types and analyzes effects of image scale, word length, speaker pose, and model capacity.The reported analyses are used to examine consistency and challenges in naturally distributed data.
II. RELATED WORK
Prior word-level lip-reading datasets range from small, controlled collections to larger datasets with natural speech conditions. Mandarin remains underrepresented, while existing benchmarks retain constraints in scale, viewpoint, duration, or sample distribution.
- Existing datasets: Earlier AVICAR and AVLetters datasets used words formed from digits and letters and supplied an initial basis for automatic lipreading research.
- Existing datasets: OuluVS provides 10 phrases and 817 sequences, averaging only 81.7 samples per class, which limits coverage of practical conditions.
- Existing datasets: OuluVS2 adds five fixed viewpoints and 53 subjects, but offers few variations beyond those view conditions.
- Existing datasets: LRW contains 500 classes and more than a thousand speakers with diverse TV-derived speech conditions, but its words and classes have roughly equal durations and sample counts.
- Mandarin gap: Few Mandarin lip-reading datasets were publicly available, motivating LRW-1000 as a contribution toward filling this gap.
B. Lip reading Methods
Lip-reading methods differ mainly in how their front ends represent spatial and temporal information. The paper compares fully 2D, fully 3D, and mixed 2D–3D convolutional approaches on LRW-1000.
- Fully 2D CNN based: Fully 2D CNN methods process frames individually and combine their representations across the sequence.A representative multi-tower structure uses single frames or frame stacks as inputs and concatenates tower activations.
- Fully 3D CNN based: Fully 3D CNN methods transform raw video into spatio-temporal features before recurrent layers generate the transcription.LipNet is described as using three 3D convolutional layers followed by gated recurrent units.
- Mixture of 2D and 3D convolution: Mixed 2D–3D methods combine spatial feature extraction from 2D convolutions with temporal modeling from spatio-temporal convolutions.One cited approach combines a spatio-temporal convolutional layer with a 2D residual network and reaches 83% accuracy on LRW.
- Comparison: The benchmark evaluates these three front-end categories to compare currently popular lip-reading approaches.
III. DATA CONSTRUCTION
LRW-1000 is constructed from diverse Mandarin television programs through a pipeline combining automated detection and transcription with manual annotation and filtering. The collection preserves broad variation in speakers, sources, resolutions, and imaging conditions.
- Pipeline: The dataset pipeline combines automated processing, manual annotation, and extra filtering to produce research-consistent clips.The pipeline begins with television-program selection and continues through data preprocessing.
- Source collection: Twenty-six broadcast sources covering 51 programs provide more than 500 hours of news and conversational Mandarin or dialectal Chinese content.Sources include regional and national stations with presenters, guests, reporters, and interviewees.
- Source collection: Daily IPTV recordings include 1920 × 1080 high-definition and 1024 × 576 standard-definition video, creating a wide range of input scales.Recordings use 25 fps H.264 video with variable video and audio bitrates.
- Shot and face processing: Shot boundaries are detected from adjacent-frame color histograms, after which three evenly spaced frames are used for face detection.Shots without a face larger than 20 × 20 pixels are dismissed as lacking potential speakers.
- Shot and face processing: A low 20 × 20-pixel candidate-face threshold supports an in-the-wild setting, although few samples have lip resolution below 20 × 20.
C. Annotations, Face Detection, and Face Tracking
The annotation and tracking pipeline combines rough automatic transcripts with manual clip annotation, face association, tracking, and synchronization validation. SyncNet corrects audio-video offsets by matching visual and audio features.
- Annotations: Rough transcripts from iFLYREC provide sentence-level timing, voiced-segment detection, and speaker diarization before manual clip annotation.Annotators record the active speaker’s position, gender, exact speech endpoints, and additional clip information.
- Face Detection and Tracking: Face association uses SeetaFaceEngine2 landmarks and coordinate comparison, followed by KCF tracking across each speaking duration.Tracking quality is automatically validated every 15 frames with a CNN-based face detector.
- Synchronization: Audio-video desynchronization occurs in some collected videos, with the largest offset below one second.The pipeline explicitly checks synchronization issues after face tracking.
- Synchronization: SyncNet searches offsets within ±15 frames by minimizing distances between 3D visual features and MFCC-based audio features.The model processes five cropped-face frames and averages distances over utterances from each video.
E. Facial Landmark Detection and Mouth Region Extraction
Mouth-region extraction aligns faces, selects a mouth-centered region, smooths its geometry, and filters inaccurate crops. The preprocessing is designed for challenging pose and scale variation in natural videos.
- Facial Landmark Detection and Mouth Region Extraction: Facial landmarks rotate faces toward horizontal eye alignment before extracting a square mouth-centered region of interest.The region size is adapted to yaw variation.
- Facial Landmark Detection and Mouth Region Extraction: A first-order Savitzky-Golay filter with window length 3 smooths rotations, center coordinates, and region-of-interest size.The filter is applied to the estimated face rotations and spatial box parameters.
- Filtering: A binary CNN removes non-lip images when large yaw or pitch causes landmark prediction to fail.Negative crops are generated by randomly shifting the initial crop region before dataset filtering and model fine-tuning.
- Filtering: LRW-1000 preserves challenging variation in scale, resolution, background clutter, and speaker attributes while providing cropped lip images.The released crops reduce the need for users to reproduce cumbersome preprocessing details.
A. Source Videos
The benchmark is built from television news and current-affairs videos, with program-level splitting and naturally varying sample properties. Its extracted corpus contains 718,018 clips spanning about 57 hours.
- Source Videos: 840 raw videos are drawn from 51 television programs, each lasting 20 minutes to 2 hours, with program-level train, test, and validation separation.This split is intended to reduce speaker overlap across subsets.
- Source Videos: 634 training videos, 84 test videos, and 122 validation videos are selected from 44, 4, and 3 programs respectively.The corresponding durations are more than 415 hours for training, 43.4 hours for test, and 48.95 hours for validation.
- Word Samples: 718,018 extracted clips span about 57 hours, with an average of about 718 samples per class and an average duration of about 0.3 seconds.Clip durations range from about 0.01 to 2.25 seconds.
- Lip Region Resolution: The benchmark retains naturally distributed input scales rather than deleting small or large sequences.Most samples have moderate sizes, while two distribution peaks arise from standard-definition and high-definition source videos.
D. Speakers
The benchmark includes more than 2,000 diverse speakers and evaluates three front-end architectures under a multi-class word-recognition setup. The data are mainly frontal-view despite unfiltered pose variation.
- Speakers: More than 2,000 speakers provide variation in age, pose, gender, accent, and personal speaking habits.Speakers include interviewers, broadcasters, program guests, and others from 840 videos.
- Baseline Methods: The word-level task is formulated as multi-class recognition and compares fully 2D, fully 3D, and mixed 2D-plus-3D CNN front ends.The comparison targets popular architectures under a common recognition setting.
- Speakers: The pose analysis measures pitch, yaw, and roll in angle degrees, with roll removed during preprocessing.Although pose was not deliberately filtered, the benchmark remains mainly composed of frontal views.
- Baseline Methods: The evaluated models are LSTM-5, 3D+2D, and D3D, with D3D introduced after the original LipNet failed to converge consistently.D3D transforms a 2D DenseNet into a 3D convolutional front end.
- Baseline Methods: All three models use the same back end containing a two-layer bidirectional RNN with bidirectional GRU units.This shared structure supports side-by-side comparison of the front ends.
B. Experimental Settings
The experiments standardize preprocessing and training across models, using grayscale normalized inputs, fixed cropping, Adam optimization, LRW pretraining, and dropout. Recognition accuracy and Kappa coefficient serve as evaluation metrics.
- Data Preprocessing: Images are converted to grayscale, normalized, resized to 122 × 122, and cropped to 112 × 112 before model input.Training uses synchronized random crops, while validation and testing use central crops; frames are also randomly flipped during augmentation.
- Implementation: The implementation uses PyTorch and trains models on four NVIDIA Titan X GPUs with 12GB memory each.
- Training Settings: Adam optimization starts at learning rate 0.001 with β = (0.9, 0.99), while all networks are pretrained on LRW.Dropout with probability 0.5 is applied to each model’s last layer during training.
- Evaluation Protocols: Recognition accuracy across 1,000 classes is the base metric, complemented by the Kappa coefficient because class sample counts vary.
C. Recognition Results
The benchmark evaluates representative lip-reading models across resolution, pose, and word-length conditions. Performance generally improves with longer samples and higher resolution, but drops sharply as speaker yaw increases.
- Evaluation Design: Models are compared across difficulty levels defined by input resolution, speaker pose, and sample length.The evaluation is designed to provide a comprehensive analysis of how these factors affect lip-reading performance.
- Performance vs. Word Length: All models perform similarly on short words, then become better and more stable as word length increases.The authors suggest that longer samples provide more context and may contain more training examples.
- Recognition Results: The results analyze benchmark behavior across word length, input scale, and pose using the corresponding figures and performance tables.
- Performance vs. Input Scales: Performance increases from hard to medium and easy input-resolution levels, then stabilizes above a certain input scale.Accuracy remains close to 30% for 1,000 classes even when test-sequence resolution is below 50 × 50.
- Performance vs. Speaker Pose: Recognition accuracy drops greatly as yaw angle increases across all three evaluated models.Large viewpoint changes create lip-region occlusion and make visual pattern learning more difficult.
VI. CONCLUSIONS
The paper concludes that LRW-1000 is a large-scale, naturally distributed word-level benchmark for lipreading in the wild. Its evaluation highlights scale, pose, and word-duration variation as practical challenges for current models.
- Conclusions: LRW-1000 provides a naturally distributed benchmark for evaluating lipreading under scale, pose, and word-duration variations.The paper evaluates representative lipreading methods on the dataset and provides corresponding results.
- Conclusions: These variations are described as ubiquitous in real-world applications and challenging for current lipreading models.
- Conclusions: The authors expect the benchmark and its results to inspire further research on lipreading.