Watch and Learn: Semi-Supervised Learning of Object Detectors from Videos

1. Introduction

• Large-scale video datasets lack the rich annotations of benchmark image datasets, and the human effort needed to label video at internet scale is substantial.ImageNet required 19 man-years to label boxes in 1.2 million images; YouTube receives 100 hours of video each minute.Source: §1 · p. 1
• Semi-supervised learning can expand a small labeled set, but noisy additions can cause semantic drift, motivating constraints on the learning process.The paper combines multiple weak cues, including appearance and temporal information, to constrain learning and discover diverse examples.Source: §1 · p. 1
• A naïve combination of detection and tracking performs poorly because errors in the two processes can amplify one another over time.Pure detection ignores temporal information, while pure tracking can stray over long durations.Source: §1 · pp. 1–2
• The method starts from a few sparsely labeled instances and learns additional boxes without requiring exhaustive annotation of every object in a frame.The paper describes automatically labeling hundreds of thousands of instances from a handful of labeled examples.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

• Unlike weakly supervised video methods that assume at most one dominant object and may rely on motion or appearance saliency, this approach handles multiple objects, including static instances.It requires a few sparsely labeled bounding-box examples.Source: §2 · pp. 2–3
• Tracking-by-detection methods that sample negatives around labeled objects rely on an assumption that fails when unmarked regions may contain more instances of the same category.Some alternatives avoid sampling video negatives by using additional training data, whereas this approach does not use such data and can cover categories outside standard datasets.Source: §2 · pp. 2–3
• The method targets short, reliable tracklets for discovering training examples rather than long-term tracking or associating detections across frames.It therefore has minimal need for data association.Source: §2 · p. 3
• Its setting combines sparse video annotations with no explicit negative data or pre-trained object models, unlike the assumptions described for related approaches.In this setting, unmarked regions cannot safely be treated as background.Source: §2 · pp. 2–3

3. Approach Overview

• The basic detection-and-tracking combination does not constrain semi-supervised learning enough, so the approach adds mechanisms to correct their mistakes.The paper describes detector and tracker errors as potentially coupled and amplifying over time.Source: §3 · p. 3
• Detectors trained from sparse positives can produce high-confidence false positives because unmarked regions cannot safely supply domain negatives.The method uses random internet images as negatives and detects objects on a subset of video frames.Source: Figure 3 · p. 3, §3 · p. 4
• Temporally inconsistent detections are removed using a smoothness prior, and decorrelated errors across feature spaces help filter false positives.The decorrelation principle relies on different feature representations producing different errors.Source: §3 · p. 4
• Filtered detections seed conservative short-term tracking, which identifies failures and labels new examples without relying heavily on the weak detector.The goal is reliable labeling, not tracking objects over long periods.Source: §3 · p. 4
• The method selects diverse, new boxes from reliable tracklets for incremental detector updates rather than treating redundant boxes equally or retraining from scratch.An example's relevance depends partly on whether similar examples were selected in earlier iterations.Source: §3 · p. 4
• Semi-supervised learning stops when the selection algorithm finds no good candidates to add.Source: §3 · p. 4

4. Approach Details

• The approach iteratively labels boxes from sparse initial annotations, then uses newly labeled examples to update the detector.It favors incremental updates using newly added, diverse examples rather than retraining from scratch on all boxes.Source: §4 · p. 4, §4.3 · p. 5

4.1. Detections with decorrelated errors

• uLSIF outlier removal uses labeled examples as definite inliers to filter detections in a feature space different from the detector’s, producing starting points for short-term tracking.This is intended to reduce correlated false positives arising from the domain mismatch between positive and negative training data.Source: §4.1 · p. 4

4.2. Reliable Tracking

• Tracking combines detection, appearance consistency, objectness, and optical flow, and the tracker identifies failures to limit erroneous labels.A trellis graph links bounding-box nodes across consecutive frames using dominant-motion difference, spatial proximity, and area change; the path is cut when its total cost exceeds a threshold.Source: §4.2 · pp. 4–5

4.3. Selection algorithm:

• Selection favors labeled boxes that current detectors explain poorly and that are diverse from existing detectors, using Jensen-Shannon divergence of detection signatures.At most 10 boxes are selected per iteration, and inability to find new boxes is the saturation stopping criterion.Source: §4.3 · p. 5
• Figure 4 measures ablation-method detection performance on the test set using Average Purity and Recall.Average Purity and Recall are shown on the left and right, respectively.Source: Figure 4 · p. 5

5. Experiments

• The study uses VIRAT and KITTI, compares component ablations, and evaluates scalability on about a million frames and generalization to a different dataset.The ablation starts from 21 boxes and runs for 30 iterations; the large-scale setup includes 312 VIRAT videos and 30 KITTI videos.Source: §5 · p. 5, §5.1 · p. 6, §5.2 · p. 7

5.1.1 Qualitative Results

• Across ablations, box tightness degrades over iterations; Boot and Det+Track show semantic drift, while removing selection quickly harms localization.The figure compares boxes used to train ESVMs across iterations, with the same initial positives for each method.Source: §5.1.1 · p. 6, Figure 5 · p. 7

5.1.2 ESVM Detection performance

• On the test set, our method achieves higher purity and recall; purity drops two points from iteration 40 to 45, supporting the stopping criterion.Because input videos have partial ground truth, labeling purity cannot be measured directly; detector performance on the test set serves as a proxy.Source: §5.1.2 · p. 6

5.1.3 Training on all the automatically labeled data

• LSVMs trained on our automatically labeled data outperform other automatically labeled sets and approach the performance of an LSVM trained on PASCAL VOC 2007.The comparison evaluates detection AP at IoU 0.5 on a held-out, fully annotated test set.Source: Table 1 · p. 7, §5.1.3 · p. 7
• Recall for the LSVM trained on automatically labeled data saturates after iteration 30, while the ground-truth-trained VIRAT-LSVM achieves twice the performance.The authors note that their method uses confident detections and tracklets in a high-precision, low-recall regime.Source: §5.1.3 · p. 7

5.2. Large scale experiments

• Our method maintains Average Purity while substantially increasing Recall over baselines in the large-scale experiments.The ESVMs are evaluated at each iteration on a held-out test set.Source: Figure 6 · p. 8, §5.2 · p. 8
• After 30 iterations on KITTI, our labeled boxes have a 3D pose distribution closer to ground truth than the baselines, which favor front and back views.Pose distribution is used as a proxy for the diversity of the labeled set.Source: Figure 7 · p. 8, §5.2 · p. 8

• The technique addresses multiple objects without requiring exhaustive annotations in any input frame.Source: §6 · p. 8
• Decorrelated errors, reliable tracking, and diverse selection are the proposed constraints for arresting semantic drift.Source: §6 · p. 8
• Starting from a handful of labeled examples, the approach labels hundreds of thousands of new examples that also improve object detectors.Source: §6 · p. 8