Deep convolutional filter banks for texture recognition and segmentation

• Prior material datasets simplify recognition by assuming textures fill images; this work studies material and perceptual texture attributes in clutter using a dataset derived from OpenSurfaces.Texture can provide cues about material properties and object identity, especially when shape is not useful.Source: §1 · p. 1
• FV-CNN treats convolutional layers as a filter bank and pools their responses orderlessly, supporting multi-scale input, arbitrary image sizes, and transfer without fine-tuning.The approach is presented as addressing drawbacks of fully connected CNN features for texture representation.Source: §1 · p. 2
• 81.1% accuracy on MIT indoor scenes surpasses the cited state of the art of 70.8%.The introduction also reports significant state-of-the-art improvements in texture recognition.Source: §1 · p. 2
• FV-CNN removes the domain-specific advantage reported for CNNs and matches the previous best on PASCAL VOC 2007, while boosting CNNs and nearing competitors on CUB with ground-truth boxes.The comparison also reports a measurable boost over CNNs on PASCAL VOC 2007 and a close approach to competitor methods on CUB 2010–2011.Source: §1 · p. 2
• 87.0% accuracy on MSRC segmentation exceeds the previous best of 86.5%, while R-CNN obtains 57.7%.FV-CNN computes convolutional features once per image and pools them over regions; OpenSurfaces segmentation results are described as promising.Source: §1 · p. 2
• SIFT is competitive only with early CNN layers, while deeper architectures such as Simonyan and Zisserman’s yield better texture filter banks [39].Source: §1 · p. 2

• Earlier datasets either used controlled conditions or, despite using Internet images, kept texture filling the camera’s field of view; this work studies textures in the wild and in clutter.Source: §2 · p. 2
• OpenSurfaces contains 25,357 images with high-quality texture/material segments; the material experiments focus on 53,915 segments across 22 classes in 10,422 images.The retained images are split evenly into training, validation, and test sets; segment sizes vary, with half under 64 × 64 pixels.Source: §2 · pp. 2–3
• The material dataset retains classes with at least 400 examples and merges several less common materials into an “other” pseudo-background class.Source: §2 · p. 3
• The attribute dataset adds manual labels for eleven DTD attributes, selected because each had at least 100 matching OpenSurfaces segments.The labels were applied to the 53,915 segments retained for material experiments.Source: §2 · p. 3

3.1. Region description

• FC-CNN and FV-CNN use the same ImageNet-pretrained CNN features, enabling a direct comparison of shape-oriented and texture-oriented region descriptors.Source: §3.1 · p. 3
• FC-CNN uses the penultimate fully connected layer to capture region shape, warping the region’s bounding box plus a 10% border to the CNN’s fixed input size.It is described as an object descriptor and follows the R-CNN descriptor approach [14].Source: §3.1 · p. 3
• FV-CNN densely pools local features from the last convolutional layer with Fisher Vectors, discarding global spatial information to represent texture.Features are extracted at multiple scales and pooled into one FV, without resizing the image to a fixed input size.Source: §3.1 · p. 3
• Unlike the related VLAD method on dense FC-CNN-like window descriptors [15], FV-CNN uses convolutional features, avoiding repeated network computation and improving efficiency.The paper reports that FV-CNN is also more accurate.Source: §3.1 · p. 3

3.2. Region segmentation

• The segmentation method generates candidate regions from low-level cues, then classifies them with FC-CNN or FV-CNN; overlapping pixel labels are resolved by voting.It uses crisp regions or multi-scale combinatorial grouping; the low-level cues separate image-specific texture instances, while the descriptors identify texture classes.Source: §3.2 · pp. 3–4

4. Results

• For region recognition on known supports, FV-CNN is evaluated on material, texture-attribute, object, scene, and fine-grained benchmarks, with support set to the image or annotated region as applicable.The section uses standard dataset-specific protocols and accuracy measures; the benchmark setup includes FMD, DTD, KTH-TIPS-2b, PASCAL VOC 2007, MIT Indoor, and CUB.Source: §4 · p. 4, §4.1 · p. 5, Table 1 · p. 5

4.1. Region recognition: textures

• On standard texture benchmarks, FV-CNN is within ±2% of FC-CNN on FMD and KTH, but improves DTD accuracy from 58.7% to 66.6%.The reported comparisons also show FC-CNN improving over DeCAF on FMD (60.7% →70.4%) and DTD (54.8% →58.7%).Source: §4.1 · p. 5, Table 1 · p. 5
• Combining FC- and FV-CNN improves performance by 1 −3% across texture datasets, and FC+FV-CNN-VD improves the state of the art by at least 6% in all of them.Adding D-SIFT brings at most ∼1% improvement and sometimes slightly degrades performance.Source: §4.1 · pp. 5–6, Table 1 · p. 5

4.3. Texture segmentation

• 58.1% →65.1% material accuracy on OS+R when switching FV-CNN from VGG-M to VGG-VD; with crisp regions, MSRC accuracy reaches 87.0%.The OS+R result improves on FC-CNN and D-SIFT, which each start at about 46%; the MSRC result is just above the best published 86.5%.Source: Table 2 · p. 5, §4.1 · p. 6, §4.3 · p. 7

4.2. Region recognition: objects and scenes

• 81.1% accuracy on MIT Indoor with VGG-VD FV-CNN, versus 68.1% for FC-CNN; FV-CNN also performs better than FC-CNN on CUB and CUB+R.On MIT Indoor, FV-CNN gains +5% with VGG-M and +13% with VGG-VD; the CUB+R difference is +10 −14%.Source: Table 1 · p. 5, §4.2 · p. 6
• FV-CNN outperforms FC-CNN across the reported cases, with gains up to 20% under ImageNet-to-MIT Indoor transfer; its performance is less sensitive to domain shifts.The authors offer as a tentative explanation that convolutional layers are less committed to a specific dataset than fully connected layers.Source: Table 3 · p. 6, §4.2 · p. 7

4.3. Texture segmentation

• 55.7% accuracy on OS with overlapping MCG proposals; performance is comparable to crisp regions, while non-maximum suppression performed poorly in initial attempts.The authors suggest material information is localized and irregularly shaped; combining evidence from multiple segmentations remains an opportunity for improvement.Source: Table 2 · p. 5, §4.3 · p. 7

4.4. Convolutional layer analysis

• CNN filter banks from VGG-M layer 3 onward significantly outperform dense SIFT, and performance improves monotonically from layer one to five.The analysis compares filter banks across CNN layers and datasets, including their receptive-field sizes and FV dictionary sizes.Source: §4.4 · p. 7, Figure 4 · p. 7

5. Conclusions

• Orderless pooling of CNN features proved an effective texture descriptor and versatile scene and object descriptor across the paper’s benchmarks.The experiments include material and texture-attribute recognition in a large dataset of textures in clutter derived from OpenSurfaces.Source: §5 · p. 7
• Figure 2 compares FC-CNN and FV-CNN material-recognition segmentations and error maps on an OpenSurfaces example.The ground truth does not annotate all pixels, and errors are restricted to annotated pixels.Source: Figure 2 · p. 8
• Figure 3 presents MSRC object-segmentation examples with ground truth, FC-CNN and FV-CNN results, and errors.Source: Figure 3 · p. 8