Deep Feature Factorization For Concept Discovery

• DFF applies non-negative matrix factorization to deep CNN activations to find semantic correspondences within and across images, requiring only input images for discovery.Annotated data is used for evaluation only.Source: §1 · pp. 1–2
• DFF heat maps correspond to semantic concepts, including object parts and hierarchical decompositions, even for classes unseen by the CNN.The paper describes clusters for bodies, limbs, arms, and legs.Source: §1 · p. 2
• DFF heat maps can support co-localization and co-segmentation of objects and parts, including distinct parts within a shared object.The method is described as weakly supervised because it uses a pre-trained CNN.Source: §1 · p. 2
• The paper introduces DFF as a tool both for insight into CNN representations and for localizing objects and object parts.Experiments cover several datasets and CNN architectures.Source: §1 · p. 2

2.1 Localization with CNN Activations

• Unlike heat maps tied to designated output units and classes, DFF factors CNN features into heat maps of common activation patterns, enabling localization of objects without a relevant output unit.The illustration describes reshaping the resulting k factors into k heat maps.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2 CNN Features as Part Detectors

• Unlike prior part-detector analyses that require ground-truth labels to identify matching units, DFF finds parts without labels and uses labeled data only for evaluation.Prior analyses measured overlap between feature activations and part-level ground truth.Source: §2.2 · p. 3

2.3 Non-negative Matrix Factorization

• Although NMF has been applied across domains and extended in several ways,,,, [9],, the authors report its use on pretrained neural-network activations as new.Source: §2.3 · p. 3

3.1 CNN Feature Space

• CNN feature maps are treated as points in a shared feature space, where deeper-layer features are described as more abstract and semantically meaningful.Each spatial position represents a pixel patch using a channel-dimensional feature vector.Source: §3.1 · p. 4

3.2 Matrix Factorization

• PCA’s positive and negative components can cancel, whereas NMF’s additive factors are intended to be more interpretable.NMF produces a user-defined rank-k approximation with non-negative H and W.Source: §3.2 · pp. 4–5

3.3 Non-negative Matrix Factorization on CNN Activations

• CNN activations are non-negative after ReLU, so they can be factorized after flattening spatial positions into rows and concatenating features across images.The resulting factorization can include images of different sizes.Source: §3.3 · p. 5

3.4 Interpreting NMF Factors

• Each DFF factor is a feature-space centroid corresponding to a coherent object or object-part, and its coefficients reshape into heat maps of matching regions.The heat maps are upsampled with bilinear interpolation to match the input image size.Source: §3.4 · p. 5, The W Matrix · p. 6, The H Matrix · p. 6

4.1 Implementation Details

• The implementation uses NMF optimization with multiplicative updates, while experiments consider VGG-16 and VGG-19 with and without batch-normalization, as well as ResNet-101.On an NVIDIA Titan X, the implementation processes over 6K images of size 224×224 at once with k = 5 and requires less than a millisecond per image.Source: §4.1 · p. 6, Neural Network Models · p. 6

4.2 Segmentation and localization methods

• The experiments distinguish pixel-wise segmentation from bounding-box localization, and single-image segmentation from jointly segmenting images into corresponding regions.Object co-segmentation separates common foreground from background, whereas part co-segmentation yields k segments for parts of the common object.Source: §4.2 · pp. 6–7

4.3 Experiments on iCoseg

• DFF produces coarse heat maps from low-resolution deep CNN activations, so the experiments also refine them with a CRF post-processing step.The CRF uses bilinearly upsampled DFF heat maps as unary terms and guided filtering for the pairwise term.Source: Part co-segmentation · p. 8, Object and Part co-segmentation · p. 9
• On iCoseg, DFF factors correspond to coherent parts, and increasing k reveals finer parts nested within whole-object factors.With k = 1, DFF is compared with state-of-the-art object co-segmentation methods; with k > 1, it performs part co-segmentation.Source: Part co-segmentation · p. 8, Table 1 · p. 10, Object and Part co-segmentation · p. 10
• iCoseg factor quality varies across convolutional blocks, while layer order within a block does not strictly predict performance.The optimal k is generally between 3 and 5, varies with network, layer, and data batch, and DFF performs similarly within the VGG family but lower on ResNet-101.Source: Object and Part co-segmentation · p. 11, Figure 4 · p. 11

4.4 Object Co-Localization on PASCAL VOC 2007

• DFF achieves better average co-localization and 16% higher CorLoc for dog than previous approaches on PASCAL VOC 2007.The paper describes the method as simpler and more general; Table 2 reports the co-localization results.Source: §4.4 · p. 12, Table 2 · p. 12

4.5 Part Co-segmentation in PASCAL-Parts

• On PASCAL-Parts cow and horse, weakly supervised DFF compares favorably with fully supervised methods and surpasses them in one case.DFF uses no hand-crafted features; factors were semi-automatically mapped to labels by examining five of approximately 140 images.Source: Evaluation · p. 12, Table 3 · p. 13
• DFF’s k = 4 factor localizes people appearing in 60% of motorbike images, distinguishing them within a set of common objects.Some factors also divide parts more finely than the ground-truth annotations, such as separating a wheel into top and bottom.Source: Evaluation · p. 13, Table 4 · p. 13

5 Conclusions

• DFF reveals hierarchical clusters in CNN feature space that correspond to part-based decompositions at multiple levels of granularity.These structures are described as part clusters nested within object clusters.Source: §5 · p. 14
• DFF achieves results on challenging co-segmentation and co-localization benchmarks that are on par with state-of-the-art methods.Source: §5 · p. 14
• DFF distinguishes parts within a common object, such as matching head to head and torso to torso, and can support semi-automatic image labeling.Source: §5 · p. 14