Source-linked AI summary
Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data
Hilde Kuehne, Ahsan Iqbal, Alexander Richard, Juergen Gall
TL;DR
Fully supervised action recognition depends on curated, preclipped data, while real-world learning of arbitrary classes remains a challenge. The paper builds a cooking-video test set and mines training examples from YouTube subtitles, then uses a semantic hierarchy to address inconsistencies in the mined classes. The authors report that modeling semantically related actions at different granularities can significantly improve performance in noisy scenarios.
Problem
Fully supervised action recognition relies on preclipped videos or temporal labels, which require human annotation and constrain the scale and real-world class coverage of datasets.
Source: §1 · p. 1Method
The paper densely annotates 250 cooking videos across 512 classes and mines about 200,000 training samples from YouTube subtitles, then organizes mined classes in a semantic hierarchy.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The dataset provides the data and fixed test set for reproducible evaluation of action learning from subtitle-mined videos.
Source: §1 · p. 2Automatically mined labels can be semantically inconsistent, including visual homonyms such as “egg” referring to different preparation states.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
Action recognition is so far mainly focusing on the problem of classification of hand selected preclipped actions and reaching impressive results in this field. But with the performance even ceiling on current datasets, it also appears that the next steps in the field will have to go beyond this fully supervised classification. One way to overcome those problems is to move towards less restricted scenarios. In this context we present a large-scale real-world dataset designed to evaluate learning techniques for human action recognition beyond hand-crafted datasets. To this end we put the process of collecting data on its feet again and start with the annotation of a test set of 250 cooking videos. The training data is then gathered by searching for the respective annotated classes within the subtitles of freely available videos. The uniqueness of the dataset is attributed to the fact that the whole process of collecting the data and training does not involve any human intervention. To address the problem of semantic inconsistencies that arise with this kind of training data, we further propose a semantical hierarchical structure for the mined classes.
1. Introduction
The paper addresses the limits of fully supervised action datasets by building a real-world benchmark with subtitle-mined training data and a semantic hierarchy for the mined classes.
1. Introduction
- The benchmark densely annotates 250 cooking videos, each several minutes long, with realistic action labels, yielding 512 action classes and 10,412 labeled samples overall.Source: §1 · p. 1
- Hand-supervised datasets require preclipped actions or temporal labels, making large-scale collection costly and limiting class coverage for real-world applications.The authors note that manually compiling training data for the benchmark’s many classes would be time-consuming.Source: §1 · p. 1
- Subtitle-based YouTube searches yield about 200,000 training samples from 21,056 videos without human intervention.The mined classes are based on class-related attributes in transcripts of the audio stream.Source: §1 · p. 2
- Semantically related classes in the hierarchy significantly improve classification results in the mined-data setting.The hierarchy allows each node to be trained independently, unlike structures that only define a chain for leaf nodes.Source: §1 · p. 2
2. Related Work
Earlier action datasets generally use queried and clipped web videos or annotate segments in long videos. Prior subtitle- and script-based work offers related approaches, but differs in its supervision, reproducibility, or scope.
2. Related Work
- Web-sourced datasets typically search predefined action queries and clip videos, while long-video datasets label action segments for temporal detection or segmentation.Examples include HMDB, UCF, and Kinetics for queried clips, and THUMOS, MPI Cooking, Breakfast, and Epic Kitchen for long-video segments.Source: §2 · p. 2
- Malmaud et al. [20] combine subtitles, audio, and vision for cooking videos but evaluate manually, while Alayrac et al. [1] require fixed action order and human video preselection.The described Alayrac et al. dataset contains five tasks, 150 videos, and about 800,000 frames.Source: §2 · p. 2
- Related webly supervised methods use transcript-to-video mapping [2], semantic transfer [15], WikiHow instructional videos [25], or learned labeling policies [32].Other cited work applies web-video learning to semantic segmentation [12].Source: §2 · p. 3
3. Mining actions without annotation
The section describes building a cooking-action dataset from annotated test videos and subtitle-mined training clips, then documents limitations and challenges in the mined data.
3.1. Test data annotation
- The 250-video test set covers five egg dishes; dense frame labels use 900 verb–object candidates and prioritize audio descriptions, yielding 512 classes.The retained classes are based on 94 verbs and 171 objects.Source: §3.1 · p. 3
3.2. Mining training data
- Training-video searches combine class or object queries with retrieved video titles and recipe titles, yielding 130K URLs before duplicate removal.The available captions came from 36,399 unique videos: 5,842 manually captioned and 30,557 automatically captioned.Source: §3.2 · p. 3
- Automatically generated captions lack punctuation and use fixed 5–7-second segments, which can separate action mentions from execution and obscure action duration.This complicates sentence parsing and can increase the chance of missing an action mentioned before it occurs.Source: §3.2 · p. 4
3.3. Subtitle parsing
- Subtitle matching ranges from adjacent verb–object pairs to ordered keyword lists and unrestricted keyword combinations, with the last strategy allowing multiple labels per clip.The authors applied scrambling because it produced the largest number of instances and more correctly detected frames than the other strategies.Source: §3.3 · p. 4, Figure 2 · p. 4
- Scrambling produced 288k action instances over 52M frames and more correctly detected frames than the two alternative mining strategies.The strategies were evaluated using detection counts, correctly detected frames, Jaccard index, and frame-based hitrate.Source: §3.3 · pp. 4–5, Table 1 · p. 4
- Neutral subtitles without dictionary keywords supply background samples, which make up 10% of the training set.Source: §3.3 · p. 5
3.4. Discussion
- The training distribution has a long tail: 102 classes occur fewer than 50 times, while the ten most frequent non-background actions constitute 25.54% of training data.The authors identify the many low-sample classes as a potential resource for one-shot and zero-shot learning.Source: §3.4 · p. 5
- Mined labels include visual homonyms, linguistic polysemy, and synonyms, such as different meanings of “pepper” and equivalent descriptions like “add” and “pour.”Egg labels can also refer to whole, liquid, or prepared states.Source: §3.4 · p. 5
4. Hierarchical modeling
The hierarchy organizes mined action classes by relatedness and level of abstraction, from general actions near the root to specific labels at the leaves. Meta classes group actions of similar granularity, while the tree also reflects differences in sample frequency.
4. Hierarchical modeling
- The hierarchy places abstract action concepts near the root and increasingly specific labels, such as “cut basil,” at the leaves.General-object actions such as “mix it” sit closer to the root.Source: §4 · p. 5
- Twenty meta classes, such as “cut meta,” parent related-granularity actions including “cut it,” “chop it,” and “slice it.”Verbs form early branches, followed by object-abstraction levels; Figure 4 illustrates the “grease meta” branch.Source: §4 · p. 6
- The model is a tree with one parent per node, multiple possible children, and a maximum depth of 18 levels.Root-near nodes tend to have more samples, while specific leaf classes tend to have fewer.Source: §4 · p. 6
5. Evaluation
The evaluation extracts frame-wise features using a pretrained two-stream model and tests temporal alignment of video frames to ordered action classes. It reports mean per-video IoU and IoD, with IoU treated as the primary measure.
5.1. Feature computation
- Features come from the spatial and temporal streams of pretrained BNInception TSN, using center crops and every second frame without task-specific fine-tuning.The authors use the last global-pooling layer outputs and cite the TSN framework [31].Source: §5.1 · p. 6
5.2. Task description
- Temporal alignment assigns video frames to a supplied ordered action list, with the transcript and video available at test time.Predefined test classes reduce false synonym-label confusions such as “add oil” versus “pour oil.”Source: §5.2 · p. 6
- For each video, IoU and IoD are computed and averaged across videos; IoU is the primary measure because IoD can improve when a class such as background becomes larger relative to other classes.Source: §5.2 · p. 6
5.3. Baselines
The baselines compare random and uniform temporal alignment with labels obtained by searching test subtitles for class keywords. Uniform alignment scores slightly higher than random alignment on both reported measures, while subtitle-only alignment has lower IoU.
5.3. Baselines
- 5.45 IoU is the uniform-alignment score, compared with 5.27 for random alignment.Uniform alignment distributes segments evenly across the video; random alignment sets boundaries randomly and averages ten runs.Source: §5.3 · p. 6
- 9.12 IoD is the uniform-alignment score, compared with 9.11 for random alignment.Both methods assume the ordered action classes are known at test time.Source: §5.3 · p. 6
- 3.90 IoU and 8.35 IoD are the scores for alignment using subtitle keyword appearances alone.This baseline uses textual input without further training.Source: §5.3 · p. 6
5.4. Model evaluation
The model evaluation trains MLP and GRU systems on a 100k-sample subset and aligns frame predictions to transcripts using Viterbi decoding. The GRU performs worse than the MLP, with early overfitting on the noisy data.
5.4. Model evaluation
- Both architectures are trained for 15 epochs on a 100k-sample subset, with the GRU input limited to chunks of 10 frames.The learning rate starts at 0.1 and drops by a factor of 0.1 every five epochs.Source: §5.4 · p. 7
- At test time, class-conditional probabilities are decoded with Viterbi to align frame predictions to the supplied transcripts.The system hypothesizes boundaries every 30 frames and caps action segments at 1000 frames.Source: §5.4 · p. 7
- The GRU performs lower than the MLP, and its greater complexity leads to early overfitting on the highly noisy training data.The authors report that this overfitting reduces overall classification accuracy.Source: §5.4 · p. 7
5.5. Comparison of spatial and temporal stream
The spatial and temporal streams alone perform only slightly better than random, while both appearance and motion can be crucial for recognizing these actions.
5.5. Comparison of spatial and temporal stream
- Each single stream performs only slightly better than the random baseline.Source: §5.5 · p. 7
- Both appearance and motion can be crucial because actions differ in which cue defines them.Source: §5.5 · p. 7
- “Cut tomato” is appearance-driven, whereas “whisk egg” is more defined by the executed action.Source: §5.5 · p. 7
5.6. Comparison to webly mined video data
Subtitle-mined instances outperform training on whole web-crawled videos labeled by search terms. Doubling the mined sample count improves IoU by more than 2%, though rare classes remain underrepresented.
5.6. Comparison to webly mined video data
- Vanilla webly supervised training on whole web-crawled videos performs much worse than training on subtitle-mined instances.Source: §5.6 · p. 7
5.7. Comparison of subset vs large-scale data
- More than 2% IoU improvement follows when the training set grows from 100k to 200k samples.Source: §5.7 · p. 7
- Even with 200k samples, the number of examples for rare classes remains insufficient to significantly increase overall performance.Source: §5.7 · p. 7
5.8. Hierarchical inference
Hierarchical consensus inference improves overall accuracy, with the best results from combining top-down and bottom-up strategies. The authors interpret this consensus as regularization for noisy training data.
5.8. Hierarchical inference
- Combining top-down and bottom-up inference increases IoU by over 3.5% and IoD by over 12%.Both strategies improve overall accuracy, with their combination producing the best results.Source: Table 8 · p. 8, §5.8 · p. 8
- The tested approaches that incorporated hierarchy during training did not outperform consensus voting.Source: §5.8 · p. 8
- Top-down inference multiplies a node's probability by its parents' probabilities; bottom-up inference aggregates child probabilities along the path.The two inference results are combined by taking their mean.Source: §5.8 · p. 8
- The authors interpret consensus as regularizing noisy data because related classes must respond similarly, which random noisy clips are less likely to do.Source: §5.8 · p. 8
6. Conclusion
The paper presents a dataset for learning action classes from unannotated videos, mined automatically from subtitles without human intervention. Its hierarchical structure models semantically similar actions at different granularities and can improve performance in noisy scenarios.
6. Conclusion
- The proposed dataset targets learning action classes from unannotated videos.Source: §6 · p. 8
- Subtitle-based mining, using human-edited and automatically generated subtitles, extracts action classes and trains them without human intervention.Source: §6 · p. 8
- A hierarchical tree models semantically similar actions at different granularities and can significantly improve performance in noisy scenarios.Source: §6 · p. 8
- The authors suggest subtitle mining may help close the gap between handcrafted datasets and real-world training scenarios.Source: §6 · p. 8