Learning visual groups from co-occurrences in space and time

1 INTRODUCTION

• Without expert annotations, it remains unclear how representations such as objects and scenes can be discovered, or whether objects arise from environmental statistics or human-specific constructs.The paper contrasts this open question with the effectiveness of supervised object and scene classifiers.Source: §1 · p. 1
• Because adjacent locations are usually semantically related and distant ones more often distinct, spatial and temporal dependencies may reveal semantic relatedness.Source: §1 · p. 1
• A deep network predicts whether two images or patches are likely to be nearby in space or time, using the learned dependencies to uncover visual groups from unlabeled input.The framework is applied to object proposals, movie scene segmentation, and grouping semantically related photographs.Source: §1 · p. 1

2 RELATED WORK

• Perception research links statistical structure and co-occurrence to perceptual groups, including proposals that infants learn linguistic groups from phoneme co-occurrence.The cited examples include Barlow (1985), Saffran et al. (1996), and Fiser & Aslin (2001).Source: §2 · p. 2
• Computer vision commonly models affinities before grouping elements; this work follows that pattern but uses spatial and temporal dependence rather than hand-engineered cues or direct supervision.Figure 2 illustrates the paper's patch-affinity and spectral-clustering pipeline.Source: §2 · p. 2, Figure 2 · p. 3
• Prior co-occurrence methods use generative probability models, which the authors say restrict scaling to high-dimensional data; their discriminative model is described as easily scalable.The cited examples include Sivic et al. (2005) and Isola et al. (2014).Source: §2 · p. 2
• Self-supervised representation learning predicts aspects of raw sensory data from spatial or temporal structure, while this paper seeks explicit, semantically meaningful visual groups instead of generic features.The related work includes predicting future frames and relative patch orientation; Doersch et al. (2015) is cited for the latter.Source: §2 · pp. 2–3

3 MODELING VISUAL AFFINITIES BY PREDICTING CO-OCCURRENCE

• The method treats spatial or temporal proximity as a proxy for shared semantic labels and defines affinity as the probability that two primitives co-occur in context.Affinity is used for clustering, including with spectral clustering.Source: §3 · p. 3

3.1 PREDICTING CO-OCCURRENCES WITH A CNN

• A shared-weight Siamese CNN compares two primitives to predict proximity, trained with logistic loss on 500,000 examples split evenly between positive and negative pairs.The experiments cover image patches, video frames, and geospatial photos.Source: §3.1 · p. 4

3 MODELING VISUAL AFFINITIES BY PREDICTING CO-OCCURRENCE

• Across image patches, video frames, and geospatial photos, the co-occurrence classifier matches or outperforms baseline grouping cues on adjacency and semantic-sameness tasks.Table 1 reports average precision for predicting C and Q across the three domains.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 PATCH AFFINITIES FROM CO-OCCURRENCE IN IMAGES

• AP 0.80: a network trained only on patch adjacency predicts same-object labels, while direct training on those labels reaches AP 0.85.The direct-label network has lower adjacency prediction performance (AP 0.92); the paper identifies the remaining semantic-sameness gap as motivation to improve context functions.Source: §3.2 · p. 5

3.3 FRAME AFFINITIES FROM CO-OCCURRENCE IN MOVIES

• AP 0.95: the frame network predicts temporal adjacency; it reaches AP 0.67 for whether two frames belong to the same DVD chapter when adjacency is held constant.The temporal-adjacency task defines positive pairs as at least 3 and no more than 10 seconds apart.Source: §3.3 · p. 6

3.4 PHOTO AFFINITIES FROM GEOSPATIAL CO-OCCURRENCE

• AP 0.79: geospatially trained photo affinities predict shared place categories, matching HOG similarity on that task.The semantic test uses eight LabelMe Outdoors categories; the cue-learning network is trained on five cities and tested on three held-out cities.Source: §3.4 · p. 6

3.5 WHICH CUES DID THE NETWORKS LEARN TO USE?

• Cue sensitivity varies by domain: patch and frame affinities depend strongly on color, while geo-photo affinities change more under geometric than color transformations.The probing tests transform B while leaving A unchanged and compare each output with the unmodified input.Source: §3.5 · pp. 6–7, Table 2 · p. 6

4 FROM PREDICTING CO-OCCURRENCE TO FORMING VISUAL GROUPS

• Across domains, the pipeline defines primitives and co-occurrence, trains a CNN to predict P(C|A,B), and uses graph edge weights for spectral clustering.Source: §4 · p. 7

4.1 FINDING OBJECTS

• Predictably co-occurring patches usually belong to the same object, motivating their grouping as object proposals that localize image objects.Source: §4.1 · p. 7
• 17 × 17 patches sampled every 8 pixels connect at 17–33-pixel center distances, with weights w(Ai,Bi)^20, then spectral clustering and diverse k-means proposals.Proposals use k values from 5 to 16; redundant proposals are pruned and rankings encourage diversity across k before random restarts at the same k.Source: §4.1 · pp. 7–8
• Without labeled-image training, object proposals are competitive with recent supervised methods for up to around 100 objects per image.The bounding-box comparison uses ABO and J metrics; implementation takes about 4 seconds per image on a 2015 Macbook Pro.Source: §4.1 · p. 8, Figure 5 · p. 8

4.2 SEGMENTING MOVIES

• Movie segmentation connects frames within ten seconds, weights edges by learned frame associations, and partitions the graph with spectral clustering.Source: §4.2 · p. 9
• The approach finds more movie boundaries with higher precision than the baselines, except color histogram similarity, which performs similarly.The recovered segmentation can mark multiple scenes within a single DVD chapter, over-segmenting relative to chapter boundaries.Source: Figure 6 · p. 8, §4.2 · p. 9

4.3 DISCOVERING PLACE CATEGORIES

• 59% purity on LabelMe Outdoors shows the learned affinities can discover semantic place categories; training used the independent MIT City dataset.The experiment clusters photos into coherent place types, with purity evaluated across values of k.Source: §4.3 · p. 9, Figure 7 · p. 9
• For photo clustering, a fully connected graph over a collection is weighted by geospatial associations before spectral clustering partitions it.Source: §4.3 · p. 9

• A general approach learns visual groupings without predefined labels, using co-occurrence in space or time as supervision.Source: §5 · p. 9