Source-linked AI summary
COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, Jie Zhou
TL;DR
Existing instructional-video datasets are limited in diversity and scale, motivating COIN’s broader benchmark for organizing and analyzing everyday task videos. COIN combines a hierarchical dataset, efficient temporal step annotation, and task-consistency refinement, with experiments showing challenging localization and improved performance. Its annotation pipeline is constrained by the computational and storage demands of improved Dense Trajectory features.
Problem
Existing instructional-video datasets are often domain-specific and too small for diverse real-world activities and data-driven learning.
Method
COIN builds a hierarchical dataset with annotated step boundaries and applies task-consistency refinement to proposal-based step localization.
Results
COIN benchmarks remain challenging, while task-consistency consistently improves R-C3D and SSN step-localization performance over their original models.
Takeaways & Limitations
COIN provides a broad benchmark for instructional-video analysis and a method for using task-level dependencies during step localization.
Takeaways & Limitations
The improved Dense Trajectory representation requires substantial computation and storage.
Abstract
from arXiv · showhide
There are substantial instructional videos on the Internet, which enables us to acquire knowledge for completing various tasks. However, most existing datasets for instructional video analysis have the limitations in diversity and scale,which makes them far from many real-world applications where more diverse activities occur. Moreover, it still remains a great challenge to organize and harness such data. To address these problems, we introduce a large-scale dataset called "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated effectively with a series of step descriptions and the corresponding temporal boundaries. Furthermore, we propose a simple yet effective method to capture the dependencies among different steps, which can be easily plugged into conventional proposal-based action detection methods for localizing important steps in instructional videos. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on the COIN dataset under different evaluation criteria. We expect the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community.
1. Introduction
Instructional-video datasets face limited diversity and scale, while organizing step-level knowledge remains difficult. COIN addresses these gaps with a hierarchical, broadly annotated dataset and a task-consistency method for step localization.
- Motivation: Existing instructional-video datasets often focus on one domain and are too small for diverse real-world activities and data-driven learning.They commonly provide step descriptions and temporal boundaries, but their domain coverage and scale remain limited.
- COIN: The dataset uses a toolbox to annotate step descriptions together with their temporal boundaries.These annotations identify where different steps occur within instructional videos.
- Method: The task-consistency method infers a whole-video task label and refines proposal scores using dependencies among steps.It is designed to plug into proposal-based action detection methods for step localization.
- Evaluation: Experiments establish COIN as a benchmark and report both substantial localization challenges and effectiveness of the proposed method.The paper evaluates multiple approaches under different criteria.
2. Related Work
Instructional-video analysis spans localization, segmentation, summarization, captioning, and grounding, supported by datasets and supervised or weakly supervised methods. COIN targets limitations in existing dataset diversity and scale while modeling dependencies among instructional steps.
- Tasks: Instructional-video analysis includes step localization, action segmentation, procedure segmentation, dense video captioning, and visual grounding.Step localization identifies step intervals and labels, whereas action segmentation assigns frame-level actions.
- Datasets: Related datasets include action detection, video summarization, and video captioning benchmarks with temporal, frame, or sentence-level annotations.COIN is related because it localizes temporal boundaries of important steps for completing tasks.
- Dataset gaps: Existing instructional datasets commonly concentrate on specific activities such as cooking and remain limited in diversity and scale.This restricts their coverage of the varied instructional activities represented in real-world use.
- Methods: Instructional-video methods are grouped into unsupervised, weakly supervised, and fully supervised learning approaches.Unsupervised approaches may combine video with narration or subtitles, while weakly supervised approaches use models such as HMMs, grammars, or Viterbi decoding.
- COIN approach: COIN’s step-localization approach further explores dependencies among different steps associated with a task.This distinguishes its focus from conventional action-detection methods described in related work.
3. The COIN Dataset
COIN organizes everyday instructional videos through a three-level lexicon and supports efficient annotation with complementary frame and video modes. Its 11,827 videos cover 180 tasks, with balanced minimum task coverage and detailed duration statistics.
- Overview: COIN is presented as a large-scale benchmark covering everyday tasks with explicit steps across a broad range of domains.The dataset is introduced through its lexicon, annotation toolbox, and statistical characterization.
- Lexicon: The lexicon uses three levels—domain, task, and step—to organize instructional videos hierarchically.The structure connects high-level activity domains to tasks and their component step sequences.
- Lexicon: COIN includes 12 domains and 180 tasks, with domain-linked tasks and expert-defined step sequences.Six experts with prior knowledge in the domains defined the steps after browsing corresponding videos.
- Annotation: The annotation toolbox combines frame mode for efficient segment selection with video mode for checking quick or consecutive actions.Frame mode samples frames at an adjustable rate, while video mode provides the online video and timeline.
- Statistics: The dataset contains 11,827 YouTube videos, split into 9,030 training and 2,797 testing samples, with more than 39 videos per task.Videos average 2.36 minutes, contain 3.91 step segments, and each segment averages 14.91 seconds.
4. Task-Consistency Analysis
The task-consistency method predicts an instructional video's task from proposal scores, then uses that task label to refine step scores before final localization. It combines bottom-up aggregation with top-down refinement to suppress task-inconsistent steps.
- Bottom-up aggregation: Proposal-based action detectors produce scores representing the probabilities of each step occurring in corresponding video proposals.
- Bottom-up aggregation: Scores are aggregated across proposals into video-level step probabilities, then combined with a binary step-task matrix to predict the video's task label.
- Top-down refinement: The method selects the predicted task's row from the step-task matrix as a mask vector for top-down score refinement.
- Top-down refinement: The refined vector attenuates weights for steps outside the predicted task, while retaining stronger weights for steps belonging to that task.The attenuation coefficient γ is empirically set to e^-2.
- Top-down refinement: Refined proposal scores are used with proposal locations in Non-Maximum Suppression to produce the final step-localization results.
5. Experiments
The experiments benchmark COIN on step localization and action segmentation, showing substantial difficulty while supporting task-consistency refinement across datasets and visualization examples.
- Experimental setup: COIN evaluates step localization and action segmentation under benchmark settings, including experiments with the proposed task-consistency method.The evaluated approaches include Random, R-C3D, SSN, and action-segmentation methods under dataset-specific supervision settings.
- Step localization: 8.12% mAP@0.5 and 26.79% mAR@0.5 are achieved by SSN-Fusion, indicating the difficulty of step localization on COIN.R-C3D+TC and SSN+TC consistently improve over their original models.
- Step localization: Task-consistency refinement discards steps inconsistent with the predicted task, producing more accurate labels in the visualization example.For “paste screen protector on Pad,” it removes “line up a screen protector with cellphone” and “open the slot of SIM card.”
- Action segmentation: 29.66% frame accuracy is achieved by TCFPN-ISBA, exceeding NN-Viterbi’s 16.23% and Action-Sets’ 4.94% under weaker supervision.CNN reaches 25.79% with frame-level supervision but remains below TCFPN-ISBA because it ignores temporal information and ordering constraints.
- Domain analysis: Sports achieves the highest domain mAP at 30.20%, while gadgets and science & craft are harder because their steps are more similar.The reported comparison uses SSN+TC-Fusion with α = 0.1 across COIN’s 12 domains.
- Cross-dataset evaluation: The task-consistency method improves SSN step localization on Breakfast, supporting its application beyond COIN.Breakfast contains over 1.9k videos, 77 hours, 4 million frames, and 48 cooking-related action categories.
- Cross-dataset comparison: COIN’s classification accuracy is 88.02%, while proposal localization is reported as almost equally challenging to YouCook2.The comparison uses TSN for classification and TAG for proposal localization.
6. Conclusions
The paper introduces COIN as a large-scale, semantically organized instructional-video dataset and proposes task-consistency modeling for step localization. Benchmark experiments show that COIN is challenging and that the proposed method is effective.
- Contributions: COIN is a large-scale instructional-video dataset organized in a rich semantic taxonomy covering broader domains and more tasks than existing datasets.The paper describes COIN as a benchmark for comprehensive instructional video analysis.
- Contributions: The task-consistency method explores relationships among different steps of a specific task.The method is evaluated alongside various approaches under different COIN scenarios.
- Conclusions: Experiments show that COIN presents substantial challenges and that the proposed method is effective.The conclusion summarizes results across the benchmark’s evaluated scenarios.
1. Annotation Time Cost Analysis
The annotation toolbox compares a newly developed frame mode with a conventional video mode. Frame-mode annotation requires substantially less time in the evaluated sample.
- Annotation time comparison: 26.8% is the frame mode’s annotation time relative to the video mode across 25 videos from 7 tasks.FM denotes the newly developed frame mode, while VM denotes the conventional video mode.
- Annotation modes: The frame mode presents extracted frames and lets annotators select segment boundaries and labels directly.Its default adjustable frame rate is 2fps, but quick consecutive actions may be missed because adjacent frames are separated in time.
- Annotation modes: The video mode provides the online video and timeline as a complementary option for capturing quick consecutive actions.The paper describes video mode as frequently used in previous work.
2. Browse Times Analysis
COIN’s selected tasks were assessed through YouTube browse times, showing substantial viewer interest across the 180-task collection.
- 1.7 × 10^8 browse times made “Make French Fries” the most-viewed task.Browse times were summed from videos appearing on the first YouTube results pages.
- 2.3 × 10^7 browse times per task was the reported average across the 180 tasks.
- The browse-time analysis was used to justify that COIN’s selected tasks meet website viewers’ needs.
3. Visualization Results
Visualization examples show that task consistency improves step localization by removing steps that do not belong to the target task.
- Task consistency discards steps unrelated to the correct task in instructional-video localization.Examples include removing “jack up the car” from “install the bicycle rack” and “crop the paper” from “make paper windmill”.
4. Step Localization Results of Different Tasks
Step localization varies across domains: visually distinct steps are easier to localize, while domains with less variation are harder.
- “Blow sugar” and “play curling” are among the easiest tasks because their steps vary substantially in appearance.They belong to the sports and leisure & performance domains, respectively.
- “Make soap” and “resize watch band” are among the hardest tasks shown in the domain comparison.They belong to the gadgets and science and craft domains, respectively.