Source-linked AI summary
Unsupervised Object Discovery and Tracking in Video Collections
Suha Kwak, Minsu Cho, Ivan Laptev, Jean Ponce, Cordelia Schmid
TL;DR
The paper addresses localizing dominant objects in noisy video collections with minimal or no supervision. It alternates cross-video discovery and within-video tracking, also revealing frame relationships across videos; it substantially outperforms prior video colocalization despite using much less supervision.
Problem
Localizing objects with little supervision remains difficult, and weakly supervised or unlabeled approaches lag strongly supervised methods.
Source: §1 · p. 1Method
The method alternates cross-video region discovery and within-video tracking while updating the frame-neighborhood structure.
Source: §1 · p. 1, §1.2 · p. 2Results
Takeaways & Limitations
It also reveals frame relationships associated with same-class instances across videos and handles multiple classes with minimal or zero supervision.
Source: §1.2 · p. 2The method is not appropriate for videos with a single dominant background and highly non-rigid objects.
Source: §6 · p. 8
Abstract
from arXiv · showhide
This paper addresses the problem of automatically localizing dominant objects as spatio-temporal tubes in a noisy collection of videos with minimal or even no supervision. We formulate the problem as a combination of two complementary processes: discovery and tracking. The first one establishes correspondences between prominent regions across videos, and the second one associates successive similar object regions within the same video. Interestingly, our algorithm also discovers the implicit topology of frames associated with instances of the same object class across different videos, a role normally left to supervisory information in the form of class labels in conventional image and video understanding methods. Indeed, as demonstrated by our experiments, our method can handle video collections featuring multiple object classes, and substantially outperforms the state of the art in colocalization, even though it tackles a broader problem with much less supervision.
1. Introduction
The paper targets localization of dominant objects in noisy video collections with minimal or no supervision. It combines cross-video discovery with within-video tracking, while iteratively updating tube selections and frame neighborhoods.
1.1. Related work
- Earlier work addresses weakly supervised or unlabeled localization, but such approaches still lag strongly supervised methods; video space-time structure may help reduce supervision.Related approaches include candidate-tube selection and image-colocalization extensions; conventional trackers often need annotations or supervised detectors.Source: §1 · p. 1, §1.1 · pp. 1–2
1. Introduction
- The method substantially outperforms prior video colocalization despite tackling a broader problem with much less supervision.It seeks spatio-temporal tubes for dominant objects in noisy collections containing multiple object classes.Source: §1 · p. 1
1.2. Proposed approach
- Discovery matches prominent regions across videos, while tracking associates object regions across successive frames within each video.Candidate regions are linked across videos through matching neighborhoods and within videos through point-track neighbors.Source: §1 · p. 1, §1.2 · p. 2
- The method alternates between selecting high-scoring tubes under a fixed neighborhood structure and updating that structure from the selected tubes.This iterative process also reveals implicit frame neighborhoods associated with same-class instances, supporting collections with multiple classes but currently only one object instance per frame.Source: §1.2 · pp. 2–3
2. Foreground confidence
Foreground confidence combines appearance- and motion-based scores to assess whether a region is an object region. Appearance measures matching across neighboring frames, while motion measures box alignment with motion clusters.
2. Foreground confidence
- Foreground confidence is a weighted sum of appearance- and motion-based confidences.Source: §2 · p. 3
2.1. Appearance-based confidence
- Appearance confidence uses region matching across neighboring frames and a standout score that compares a region’s saliency with that of its potential backgrounds.PHM scores matches using appearance and geometric consistency; standout scoring counters cases where parts match more consistently than whole objects.Source: §2.1 · p. 3
2.2. Motion-based confidence
- Motion coherence scores combine edge-bin motion-cluster labels, cluster weights, and edge-wise maximum pooling to favor boxes that contact cluster boundaries and contain clusters.The score complements appearance-based standout confidence; Figure 2 illustrates the binning, weighting, pooling, heat map, and top five regions.Source: §2.2 · pp. 3–4, Figure 2 · p. 4
3. Temporal consistency
Temporal consistency combines appearance and motion evidence to favor stable, consistent tubes despite imperfect foreground confidence or multiple same-category instances. Its motion term compares shared point tracks after mapping successive regions to a unit square.
3. Temporal consistency
- Temporal consistency addresses regions that may be misaligned by imperfect confidence or may track different instances of the same category.The term combines appearance- and motion-based evidence to make selected tubes more stable and temporally consistent.Source: §3 · p. 4
3.1. Appearance-based consistency
- Appearance-based consistency scores consecutive regions by the opposite of the distance between their HOG descriptors.Scores are rescaled to [0, 1] in practice.Source: §3.1 · p. 4
3.2. Motion-based consistency
- Motion-based consistency compares shared point-track configurations after mapping each region and its points to a unit square, allowing affine variation between regions.If consecutive regions share no point track, the method assigns θ, smaller than -1, to penalize the transition; the measure captures changes in position, aspect ratio, and scale.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
4. Discovery and tracking algorithm
The algorithm alternates between updating cross-video frame neighborhoods and optimizing object tubes within each video. It uses the localized tubes to guide later neighbor retrieval and relocalization.
4. Discovery and tracking algorithm
- The algorithm alternates neighborhood updates and per-video tube optimization from whole-video initial tubes, stopping near convergence; more than five iterations did not improve performance.It fixes k = 10 neighbors, selects each video's best tube at the end, and processes videos independently at each iteration for easy parallelization.Source: §4 · p. 5
- After initialization with GIST frame distances, neighbor retrieval uses appearance similarity from top-20 salient proposals, while relocalization considers the top 100 foreground proposals and optimizes by dynamic programming.Neighbor-frame proposals are restricted to regions in the previously localized tubes; retaining p = 5 candidate tubes per video between iterations increases robustness.Source: §4 · p. 5
5. Implementation and results
The experiments evaluate localization and video-to-video topology on the challenging YouTube-Object collection, including a fully unsupervised setting that mixes classes. Results show stronger colocalization than prior video colocalization methods and indicate that retrieved neighbors reflect both class similarity and appearance overlap.
5. Implementation and results
- The YouTube-Object evaluation uses videos from 10 PASCAL VOC classes, with ground-truth boxes available for a subset and one annotated frame per video.The collection includes large camera motion, viewpoint changes, encoding artifacts, editing effects, and incorrect shot boundaries.Source: §5 · p. 5
5.1. Implementation details
- The method processes uniformly sampled key frames at stride 20, uses long-term point tracks for motion, and interpolates regions to obtain dense localization.Key frames are used to avoid processing temporally adjacent frames with redundant information.Source: §5.1 · p. 6
5.2. Evaluation metrics
- Localization is measured by CorLoc, while unsupervised discovery uses CorRet and nearest-neighbor classification error to assess retrieval topology and localization.CorLoc uses the PASCAL overlap criterion; CorRet is the mean share of retrieved neighbors belonging to the query video's class.Source: §5.2 · p. 6
5.3. Object colocalization per class
- The full method outperforms the previous state of the art in colocalization and improves over simpler variants when temporal-consistency terms are added.Motion confidence can help for non-rigid or motion-separated objects, but can hurt when motion clusters capture only part of an object or distinctive background.Source: §5.3 · p. 6, Table 1 · p. 7
5.4. Unsupervised object discovery and tracking
- In fully unsupervised mixed-class videos, the method outperforms existing video colocalization techniques without supervisory information.The results are summarized as CorLoc scores on the YouTube-Object dataset.Source: §5.4 · p. 7, Table 1 · p. 7
- Nearest-neighbor retrieval usually returns visually similar object candidates, while classes with similar appearances show more connections and higher classification error rates.The reported examples contrast lower error for distinctive classes such as aeroplane, boat, car, and train with higher error for cat, dog, cow, and horse.Source: §5.4 · p. 7, Table 2 · p. 8
6. Discussion and Conclusion
The method combines object discovery and tracking to localize objects in unlabeled video collections, revealing relationships among frames and objects across videos. On YouTube-Object, it significantly outperforms the state of the art in colocalization with much less supervision, but is not appropriate for some videos with a dominant background and highly non-rigid objects.
6. Discussion and Conclusion
- On YouTube-Object, the method significantly outperforms the state of the art in colocalization while using much less supervision.The paper also reports that the method localizes objects in each video and reveals a network structure associating frames and objects across videos.Source: §6 · p. 8
- Alternating optimization of the localization objective and neighborhood structure improves both while combining object discovery and tracking.Source: §6 · p. 8
- Figure 8 shows nearest-neighbor retrieval as a confusion matrix, with query classes in rows, retrieved classes in columns, and diagonal entries corresponding to Table 2 CorRet values.Source: Figure 8 · p. 8
- The method is not appropriate for videos with a single dominant background and highly non-rigid objects, such as UCF-sports.Source: §6 · p. 8