PaperlayerView exact revision on arXiv

Source-linked AI summary

YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video

Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, Vincent Vanhoucke

arXiv:1702.00824v5cs.CV

TL;DR

Video object detection lacked a large, diverse, human-curated resource with frame-level classifications and tracked boxes. YT-BB supplies such a dataset through object-aware video mining and cascaded annotation, and reports high annotation quality while indicating that temporal information may improve inference.

  • Problem

    Large-scale video datasets are limited by noisy online labels, temporal redundancy, and the absence of a human-curated resource combining classification and detection annotations across diverse videos.

  • Method

    YT-BB combines object-aware YouTube video mining with staged human annotation to produce tracked bounding boxes and classification labels for natural video segments.

  • Results

    YT-BB contains 380,000 video segments and 5.6 million human-drawn bounding boxes across 23 categories, with all classes above 95% correct for boxes and absent tags under unanimous testing verification.

  • Takeaways & Limitations

    The dataset provides a large benchmark for video detection and tracking, while baseline manipulations indicate that temporal information may improve frame predictions.

  • Takeaways & Limitations

    Evaluation of the person class is imperfect because YT-BB is not exhaustively labeled and frames may contain additional unlabeled people.

Abstract

from arXiv · show

We introduce a new large-scale data set of video URLs with densely-sampled object bounding box annotations called YouTube-BoundingBoxes (YT-BB). The data set consists of approximately 380,000 video segments about 19s long, automatically selected to feature objects in natural settings without editing or post-processing, with a recording quality often akin to that of a hand-held cell phone camera. The objects represent a subset of the MS COCO label set. All video segments were human-annotated with high-precision classification labels and bounding boxes at 1 frame per second. The use of a cascade of increasingly precise human annotations ensures a label accuracy above 95% for every class and tight bounding boxes. Finally, we train and evaluate well-known deep network architectures and report baseline figures for per-frame classification and localization to provide a point of comparison for future work. We also demonstrate how the temporal contiguity of video can potentially be used to improve such inferences. Please see the PDF file to find the URL to download the data. We hope the availability of such large curated corpus will spur new advances in video object detection and tracking.

1. Introduction

Large-scale video datasets are harder to build than image datasets because online videos are weakly labeled, temporally redundant, and biased toward limited viewpoints. YT-BB addresses this gap with densely tracked, manually curated object annotations across diverse video segments.

  • Context: Image recognition and detection datasets have driven rapid progress by supplying public benchmarks and reusable systems, motivating comparable resources for video.Such systems can also transfer to other computer-vision and multimodal-learning pipelines.
  • Motivation: Weak labels in online video are noisy, making large-scale dataset construction substantially dependent on human involvement.Video data are also more difficult to scale because adjacent frames contain redundant information.
  • Motivation: Temporal redundancy means that adding sequential frames from a few videos is insufficient; datasets instead require many diverse videos.The paper emphasizes realistic lighting, occlusions, and non-canonical viewpoints as important sources of variation.
  • Gap: Natural video requires objects to be tracked across frames and precisely localized, yet no large human-curated dataset previously combined these annotations across several classes and varied videos.This gap motivates annotations designed for persistent objects in realistic scenes.
  • Contribution: YT-BB provides a large body of manually curated bounding boxes that track objects for roughly 100 frames, supporting deep-network training and visio-temporal modeling.The dataset is intended to make video learning feasible in a realistic setting.

2. Related work

Existing video datasets target actions, retrieval, or controlled tracking, while YT-BB is designed as a much larger human-annotated detection resource. Its construction combines object-aware mining with staged human labeling and quality verification.

  • Existing datasets: Existing resources include action datasets, retrieval benchmarks, small controlled tracking datasets, and limited localization sets, each differing from YT-BB in task, scale, or annotation scope.TRECVID, for example, provides 1,000 videos with boxes for 10 classes, while VOT and MOT emphasize controlled tracking difficulties.
  • YT-BB scale: YT-BB exceeds other video detection datasets by more than an order of magnitude and is described as the largest human-annotated detection dataset.The comparison is presented in Figure 1’s dataset-size chart.
  • Label design: The dataset uses 23 COCO-derived object classes, adds a NONE class for frames lacking those objects, and gives person preferential volume and mining treatment.The NONE class addresses open-world frames that do not correspond to the selected label set.
  • Data mining: Mining samples public YouTube videos using motion, view-count, camera-cut, and classifier-based filters to reduce slideshows, professionally edited clips, short scenes, and unlikely object segments.Low classifier thresholds were used to avoid selecting only easy examples.
  • Human annotations: A four-stage annotation cascade progressively labels presence, verifies frame-level classifications, draws tight boxes, and verifies boxes or absence tags.Stages 1 and 2 use multiple raters and majority voting, while later stages refine localization quality.
  • Quality control: The pipeline was finalized only after classification accuracy was estimated above 0.95 for every class, with later stages refined through disagreement-based and random feedback.Validation and testing subsets were annotated last as annotator experience increased.

4. Results

The resulting data set combines millions of annotated frames and classification tags from hundreds of thousands of YouTube videos. Its annotations achieve high verification accuracy, capture object motion, and use video-disjoint evaluation splits.

  • Data set size: 5.6 million frames from 240,000 unique YouTube videos received bounding-box annotations, alongside absent tags and millions of classification labels.The data set also includes absent detection tags in 1 million frames, 9.6 million positive classifications, and 1 million negative classifications.
  • Classification quality: Classification stages were not further optimized because their primary purpose was filtering videos for subsequent bounding-box annotation.The authors note that untrained, unvetted raters reduced classification accuracy, while retaining these annotations in the released data set.
  • Detection quality: 98% of bounding boxes and absent tags were correct for every class by majority verification, while all classes exceeded 95% under unanimous testing-set verification.For most classes, the stricter unanimous criterion still yielded at least 98% correctness.
  • Temporal characteristics: Measured box-center displacement and related statistics confirmed significant object motion across the videos.The reported statistics also include box-size changes, field-of-view entry and exit, covered area, and object presence frequency.
  • Evaluation splits: The validation and testing subsets comprise 10% of the data, preserve class proportions, and prevent any YouTube video from appearing across multiple splits.Part of the testing subset is withheld for future public challenges based on YT-BB.

5. Baseline models

The baselines compare classification and detection models trained and evaluated across YT-BB and COCO, then test whether temporal information improves frame-level predictions. YT-BB is easier for in-domain classification and detection, while cross-dataset transfer is harder and temporal smoothing has mixed effects.

  • Image classification: YT-BB and COCO models were evaluated both within dataset and across datasets using shared classification and localization labels.The comparison establishes relative task difficulty and provides baselines for future architectures.
  • Image classification: 0.93 mAP on YT-BB versus 0.83 mAP on COCO shows that classification was easier when trained and tested on YT-BB.The evaluation averages precision-recall performance across 23 classes, excluding “NONE”.
  • Image classification: 0.77 mAP for a COCO model evaluated on YT-BB versus 0.83 mAP on COCO indicates a domain-transfer penalty and diverse YT-BB imagery.The analogous transfer comparison for a YT-BB-trained model is reported in Table 1.
  • Object detection: 0.59 mAP on YT-BB versus 0.43 mAP on COCO shows that detection was easier when trained and tested on YT-BB.The result remains consistent across detection box sizes.
  • Object detection: 0.37 mAP for a COCO detection model evaluated on YT-BB versus 0.43 mAP on COCO confirms a cross-dataset detection penalty.The authors report that the analogous claim also holds for a model trained on YT-BB.
  • Exploiting temporal information in videos: Temporal smoothing leaves classification mAP and AUC largely unchanged but improves the precision-recall curve in the high-recall regime, whereas detection performance varies by class.Across all detection classes, smoothing slightly lowers mAP from 0.37 to 0.36; some classes improve while others decline.

6. Discussion

YT-BB provides a large, carefully annotated video detection resource and baselines for classification and detection. The discussion identifies hard negatives and temporal modeling as promising directions, while noting annotation-scope and scalability limitations.

  • 6. Discussion: YT-BB contains 380,000 video segments and 5.6 million human-drawn bounding boxes tracking objects in 23 categories.The annotations include tight boxes and precise absence tags, alongside baselines trained and evaluated on YT-BB and COCO.
  • 6. Discussion: Future work could add more classes, but scaling annotation would increase the challenges of maintaining label quality.The paper contrasts lower-accountability crowd-sourcing with dedicated annotators who can be trained and given feedback, while noting that no formal A/B test was conducted.
  • 6. Discussion: Exhaustively labeling objects in videos, especially in the testing subset, could improve the data despite making annotation more complex.The authors note that increasing the sheer number of boxes does not appear critical because the baselines show no signs of over-fitting.
  • 6. Discussion: Hard-negative frames, where surrounding frames contain the object but the individual frame does not, offer training and evaluation examples for future models.These frames were identified through the human annotation results.
  • 6. Discussion: The baselines suggest that temporal information can improve inferences beyond single-frame classification and detection.The paper reports potential gains from incorporating sequence information, including naive temporal manipulations.

1. Human annotation user interfaces

The annotation workflow used separate interfaces for frame-level classification and bounding-box drawing. The latter supported frame navigation, box drawing, and categorical status labels while preserving object identity across frames.

  • 1. Human annotation user interfaces: The frame-level classification tool asked annotators whether the class was present or absent for each frame.The segment-level tool displayed fewer frames and allowed one answer per segment.
  • 1. Human annotation user interfaces: The bounding-box tool let annotators navigate segment frames, draw boxes by clicking and dragging, and mark absence, uncertainty, or interface problems.An absence choice generated an absent-tag included in the data set.
  • 1. Human annotation user interfaces: Stages 3 and 4 required annotators to preserve object identity and annotate all boxes for the selected individual object.The same object had to be followed across the segment rather than switching between multiple instances.

2. Attention span of human annotators

The annotation design was adapted to limited annotator attention by putting decisive information early and simplifying each task into focused choices. A staged interface supported progressively more precise judgments.

  • 2. Attention span of human annotators: Class definitions were made operationally precise because untrained annotators could not reliably use full dictionary definitions.The paper uses concrete inclusion and exclusion rules to reduce corner cases.
  • 2. Attention span of human annotators: The airplane task asked whether a real airplane’s outside was visible, excluding interior viewpoints and uncertain cases.Annotators were instructed to answer no for uncertain examples and for views from inside the plane.
  • 2. Attention span of human annotators: Frame-level classification required a present-or-absent decision for each frame, whereas bounding-box annotation added absence, uncertainty, and interface-problem choices.These tools represent separate stages in the annotation cascade.
  • 2. Attention span of human annotators: Toy, cartoon, and video-game airplanes were explicitly excluded from positive airplane labels.Uncertain or absent cases also received a negative answer.
  • 2. Attention span of human annotators: Question wording placed most information at the beginning and delayed meaningful content until the decisive final word.The interface also used caps, bold, and bullets as attention-guiding design choices.

3. Bounding box drawing guidelines

Bounding boxes were drawn to capture recognizable visible objects consistently across video frames. Annotators used surrounding-frame context for identity but did not infer hidden or out-of-frame extent.

  • 3. Bounding box drawing guidelines: Objects were boxed when recognizable even if only a small part was visible.Recognition could rely on the object’s visible evidence and the surrounding video context.
  • 3. Bounding box drawing guidelines: Annotators could use context from other frames to identify an object that was not recognizable in the current frame.This rule connects frame-level box decisions to the temporal continuity of the segment.
  • 3. Bounding box drawing guidelines: Only the visible portion of an object was boxed, with no inference about hidden or out-of-frame parts.This constrains boxes to directly visible extent.
  • 3. Bounding box drawing guidelines: A single box included all visible parts of one object separated by an occlusion.The guideline applies when an object extends on both sides of a narrow obstruction.
  • 3. Bounding box drawing guidelines: The first box was drawn on a random positive frame, after which the annotator worked forward and backward through the segment.The starting frame was positive according to stage 2 classification.

4. Human annotation detailed statistics

The supplementary material provides detailed statistics on annotation coverage and bounding-box properties, with separate tables for class counts and object motion.

  • Supplementary Tables 1 and 2 provide complete class counts for classifications and detections, respectively.
  • Supplementary Table 3 reports quantitative measures of bounding-box size and motion.
  • The detailed statistics are provided on the following supplementary pages.

5. Relevant GitHub locations

The paper lists GitHub locations for the implementation of the evaluated models, including Inception-v3, Inception-ResNet-v2, and Faster-RCNN.

  • The Inception-v3 model is available in the TensorFlow models GitHub repository.
  • The Inception-ResNet-v2 model is available in the TensorFlow models GitHub repository.
  • The Faster-RCNN model is available in the py-faster-rcnn GitHub repository.

6. Per-class object detection baseline

The supplementary material documents annotation counts, object motion characteristics, and class-specific detection difficulty using mAP evaluations on COCO and YT-BB.

  • Human annotation counts: Supplementary Table 1 counts unique annotated frames and videos with positive or negative classification labels for each class.The “ALL” row may not equal the sum of class rows because videos can contain multiple annotated classes.
  • Human annotation counts: Supplementary Table 2 counts unique annotated frames and videos with bounding boxes or absent tags for each class.The “ALL” row may not equal the sum of class rows because videos can contain multiple annotated objects.
  • Object motion: Supplementary Table 3 averages present fraction, continuous fraction, mean area, and center RMS across segments for each class.Continuous fraction indicates how often an object enters and leaves the field of view.
  • Detection difficulty: Supplementary Table 4 measures per-class object-detection difficulty with mean average precision across precision-recall curves.Its columns report evaluation on COCO and YT-BB, respectively.
  • Detection difficulty: Supplementary Table 4 evaluates models trained on the COCO and YT-BB data sets.
Loading 1702.00824v5…