Source-linked AI summary
Mid-level Deep Pattern Mining
Yao Li, Lingqiao Liu, Chunhua Shen, Anton van den Hengel
TL;DR
Mid-level visual-element discovery seeks image-patch clusters that are both representative and discriminative, but finding discriminative patches can involve searching tens of thousands of patches. The paper combines fully connected CNN activations with association rule mining to discover patterns among patches. It reports better classification results than prior mid-level methods with fewer elements, and results that outperform or match CNN state of the art.
Problem
Mid-level visual-element discovery seeks representative and discriminative patch clusters, while finding discriminative patches can require searching tens of thousands of patches.
Source: §1 · p. 1Method
The method applies association rule mining to patterns in fully connected CNN activations of image patches.
Source: §1 · pp. 1–2Results
On scene and object classification, the approach outperforms prior mid-level visual-element methods with fewer elements and outperforms or matches CNN state of the art.
Source: §1 · p. 2Takeaways & Limitations
The paper concludes that CNN activations are well suited to pattern mining for mid-level visual-element discovery.
Source: §7 · p. 8Transactions must contain few items and record integer sets rather than real-valued features such as SIFT or HOG.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
Mid-level visual element discovery aims to find clusters of image patches that are both representative and discriminative. In this work, we study this problem from the prospective of pattern mining while relying on the recently popularized Convolutional Neural Networks (CNNs). Specifically, we find that for an image patch, activations extracted from the first fully-connected layer of CNNs have two appealing properties which enable its seamless integration with pattern mining. Patterns are then discovered from a large number of CNN activations of image patches through the well-known association rule mining. When we retrieve and visualize image patches with the same pattern, surprisingly, they are not only visually similar but also semantically consistent. We apply our approach to scene and object classification tasks, and demonstrate that our approach outperforms all previous works on mid-level visual element discovery by a sizeable margin with far fewer elements being used. Our approach also outperforms or matches recent works using CNN for these tasks. Source code of the complete system is available online.
1. Introduction
Mid-level visual elements are image-patch clusters intended to be representative of a target category and discriminative against the natural world. The paper proposes combining CNN activations with association rule mining to discover such elements and reports stronger classification performance than prior mid-level methods using fewer elements.
1. Introduction
- Association rule mining is presented as a fit for the large patch-search workload, which can involve tens of thousands of patches.The cited motivation describes association rules as discoverable efficiently from millions of transactions.Source: §1 · p. 1
- Association rule mining is proposed for discovering category-specific patterns from many image patches, using fully connected CNN activations that can be integrated with the mining algorithm.The paper identifies two properties of CNN activations that meet pattern-mining requirements.Source: §1 · pp. 1–2
- Image patches sharing a pattern are reported to be visually similar and semantically consistent, while the patterns address representativeness and discriminativeness.Representativeness means frequent occurrence in the target category; discriminativeness means visual distinction from the natural world.Source: §1 · pp. 1–2
- The approach outperforms prior mid-level visual-element methods by a noticeable margin with fewer elements, and outperforms or matches CNN state-of-the-art on scene and object classification.The evaluation trains element detectors from retrieved visual elements and uses them to generate feature representations.Source: §1 · p. 2
2. Related Work
Related work covers mid-level visual elements for image classification, CNN feature representations, and pattern mining applications in vision. These lines of work motivate the paper’s combination of CNN activations and pattern mining.
Mid-level visual element discovery
- Mid-level visual-element discovery seeks representative and discriminative image-patch clusters, used for image classification.Prior approaches include cross-validation training, validation-set ranking and selection, and discriminative mode seeking.Source: §2 · p. 2
Convolutional Neural Networks
- In the benchmark CNN architecture, five convolutional layers precede two 4096-dimensional fully connected activations, which have been used as general recognition representations.The passage also notes CNNs’ renewed popularity following results in the ILSVRC challenge.Source: §2 · p. 2
Pattern mining in vision
- Pattern mining has been applied in vision to image classification, action recognition, and human-object interaction.Its stated advantage is processing massive volumes of data.Source: §2 · p. 2
3. Properties of CNN activations of patches
The section examines whether image-patch CNN activations can be sparsified or binarized while retaining classification performance. Its conclusion is that much of the discriminative information lies in the indices of the largest activation magnitudes.
3. Properties of CNN activations of patches
- CNN-Sparsified keeps the k largest activation magnitudes and zeros the rest, while CNN-Binarized sets the k largest dimensions to one and the rest to zero; both use max pooling for image features.The experiments use 4096-dimensional activations from sampled image patches.Source: §3 · pp. 2–3
- The strategies are evaluated with one-vs-all multi-class linear SVMs, against a max-pooled CNN-activation baseline with 65.15% accuracy.The section reports that sparsified activations retain reasonably high accuracy with only k non-zero magnitudes out of 4096.Source: §3 · p. 3
- With small k, binarizing CNN activations does not harm accuracy and can slightly increase it; the findings suggest discriminative information is mostly in the indices of the k largest magnitudes.The comparison is between CNN-Binarized and CNN-Sparsified strategies; Table 1 reports their classification accuracies on MIT Indoor.Source: §3 · p. 3, Table 1 · p. 3
4. Mid-level deep pattern mining
Section 4 adapts association rule mining to discover mid-level visual patterns from CNN activations of image patches. It explains how transactions are constructed and how support and confidence thresholds yield patterns that are representative and discriminative.
4.1. Pattern mining revisited
- Support is the fraction of transactions containing an itemset, while association-rule confidence reflects how often the consequent occurs among transactions containing the antecedent.An itemset is frequent when its support exceeds a predefined threshold, and the paper seeks rules with reasonably high confidence.Source: §4.1 · pp. 3–4
- Pattern mining requires transactions with few items and integer-valued items, constraints that motivate sparse CNN activations and selecting only their largest dimensions.The paper states that the potential search space grows exponentially with transaction size; its experiments use k = 20.Source: §4.1 · p. 4, §4.2 · p. 4
4.2. Transaction creation
- Each patch becomes a transaction of the indices of its k largest CNN activation magnitudes plus a pos or neg class label.The experiments set k to 20, keeping each transaction small while retaining activation dimensions as integer items.Source: §4.2 · p. 4
4.3. Association rule mining
- Association rule mining selects patterns using support and confidence thresholds, which respectively ensure frequent occurrence in the target category and greater likelihood there than in the natural world.The paper connects these criteria to the representativeness and discriminativeness requirements for visual elements.Source: §4.3 · pp. 4–5
5. Application to image classification
The application section retrieves patches sharing mined patterns, merges redundant visual elements into detector-backed elements, then selects and pools detector responses to encode images.
5.1. Retrieving mid-level visual elements
- Patches sharing a mined pattern form a visual element retrieved through an inverted index, and the discovered elements are visually similar, semantically consistent, and invariant to horizontal flipping.The application treats each dataset class in turn as the target category and uses only training images to discover patterns.Source: §5.1 · p. 5, §5 · p. 5
5.2. Ensemble merging and training detectors
- The merging procedure greedily combines overlapping elements, trains an LDA detector, and iteratively adds elements whose patches score above a threshold until no more can be added.This reduces redundancy among elements that may overlap or describe the same visual concept; the output is a clean set of elements and corresponding detectors.Source: §5.2 · p. 5
5.3. Selecting and encoding
- To limit the computational cost of applying detectors, the method selects elements covering the most training images within each class, taking the same number from every class.Selected detector responses are evaluated at three scales, max-pooled per region in a 1 × 1 and 2 × 2 spatial pyramid, then pooled across scales.Source: §5.3 · pp. 5–6
6. Experiments
The experiments evaluate MDPM on scene and generic object classification, comparing its discovered visual elements with earlier methods and other CNN-activation approaches.
6. Experiments
- The evaluation covers 67 MIT Indoor classes, using approximately 80 training and 20 test images per class, and PASCAL VOC 2007, using training and validation data to discover patterns and train classifiers.The implementation samples 128×128 patches at a stride of 32 pixels after resizing each image's smaller dimension to 256.Source: §6.1 · p. 6, §6 · p. 6, §6.2 · p. 7
6.1. Scene classification
- 69.69% accuracy with 50 elements per class exceeds prior mid-level discovery methods; with 20 versus Doersch's 200 elements, MDPM is over 4 percentage points higher.The comparison also reports that 50 elements per class surpass RFDC by over 15% in the same setting.Source: §6.1 · p. 6, Table 2 · p. 6
- 70.46% accuracy after concatenating MDPM and whole-image CNN activations surpasses prior CNN works; MDPM alone reaches 69.69%, above MOP-CNN and SCFVC.MDPM alone is reported at 69.69% versus MOP-CNN's 65.52% with 128×128 patches and 62.24% with 64×64 patches, and SCFVC's 68.20%.Source: §6.1 · pp. 6–7, Table 3 · p. 7
- For about 0.2 million precomputed patch activations, MDPM discovers rules in about 3 minutes, versus about 9 hours for LDA-Retrained to find elements in one class.The MDPM timing includes writing the transaction file and association rule mining.Source: §6.1 · p. 7
- Discovered scene elements capture repeated category patterns, including object configurations such as a table and chair, or individual objects such as a baby cot.Figure 4 shows the elements and their detections on MIT Indoor test images.Source: §6.1 · p. 7, Figure 4 · p. 7
6.2. Object classification
- 75.2% mAP with 50 elements per class beats the global-CNN baseline's 67.3%; concatenating MDPM and global CNN features reaches 77.9%, marginally above box-annotated fine-tuning.The authors note that the fine-tuning comparison uses bounding-box annotations, which provide extra information and are time-consuming to gather.Source: §6.2 · p. 8, Table 4 · p. 8
- On PASCAL VOC 2007, discovered elements capture discriminative object parts, including dog faces.Figure 5 shows the elements and detections on test images.Source: §6.2 · p. 8, Figure 5 · p. 8
7. Conclusion
The conclusion frames mid-level visual-element discovery as a pattern-mining problem and argues that CNN activations suit this task. It connects the approach to fields where CNNs and pattern mining underpin state-of-the-art work.
7. Conclusion
- The paper addresses mid-level visual-element discovery from the perspective of pattern mining.The conclusion describes pattern mining as profitable for this task and CNN activations as particularly well suited to it.Source: §7 · p. 8
- The conclusion identifies CNN activations as particularly well suited to mid-level visual-element discovery.It presents this suitability alongside the finding that applying pattern mining to the task is profitable.Source: §7 · p. 8
- The paper connects the approach to areas where CNNs underpin state-of-the-art vision methods and pattern mining underpins state-of-the-art Big Data processing.Source: §7 · p. 8