PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning scale-variant and scale-invariant features for deep image classification

Nanne van Noord, Eric Postma

arXiv:1602.01255v2cs.CV

TL;DR

CNNs must handle image variation across resolution, object size, and scale, while scale-invariant representations alone omit scale-variant information. The paper uses a multi-scale CNN built from specialized CNNs over a Gaussian pyramid to learn and combine both types of representation. It reports that combining all scales outperforms single-scale CNNs, while acknowledging that a sufficiently complex single CNN might achieve similar performance.

  • Problem

    CNNs trained on images with varying resolution, object size, and scale need to capture task-relevant information across scales rather than relying only on scale-invariant representations.

  • Method

    The paper uses a multi-scale CNN comprising separate CNNs associated with Gaussian-pyramid scales, combining their predictions to learn scale-variant and scale-invariant features.

  • Results

    Combining all scales outperformed a single-scale CNN, and averaging class posteriors across scales produced optimal performance.

  • Takeaways & Limitations

    Combining scale-invariant and scale-variant representations benefits image recognition tasks involving image structure at varying scales and resolutions.

  • Takeaways & Limitations

    The paper cannot rule out that a substantially deeper and more complex single CNN could achieve similar performance, although such a network would require more data and computation.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) require large image corpora to be trained on classification tasks. The variation in image resolutions, sizes of objects and patterns depicted, and image scales, hampers CNN training and performance, because the task-relevant information varies over spatial scales. Previous work attempting to deal with such scale variations focused on encouraging scale-invariant CNN representations. However, scale-invariant representations are incomplete representations of images, because images contain scale-variant information as well. This paper addresses the combined development of scale-invariant and scale-variant representations. We propose a multi- scale CNN method to encourage the recognition of both types of features and evaluate it on a challenging image classification task involving task-relevant characteristics at multiple scales. The results show that our multi-scale CNN outperforms single-scale CNN. This leads to the conclusion that encouraging the combined development of a scale-invariant and scale-variant representation in CNNs is beneficial to image recognition performance.

I. INTRODUCTION

CNNs handle location and viewpoint variation relatively well, but varying image resolution, object size, and image scale makes it difficult to preserve task-relevant visual information. Resolution limits which spatial structures can be represented, while scale changes determine whether fine or coarse objects remain visible.

  • Variation in images: CNNs generally handle location variation through convolutional weight sharing and viewpoint variation through viewpoint-invariant filters, but size variation remains challenging.The challenge is especially pronounced in image collections with varying resolutions and object sizes.
  • Resolution and scale: High spatial frequencies represent fine details, while low spatial frequencies represent coarse visual structures.Spatial smoothing removes high frequencies and therefore converts a fine-scale image into a coarser-scale image.
  • Resolution and scale: Image resolution is the number of pixels sampling a visual source, whereas image scale refers to its spatial-frequency content.Fine-scale images contain both high and low spatial frequencies, while coarse-scale images contain only low spatial frequencies.
  • Aliasing: Insufficient resolution causes aliasing, distorting visual structure that cannot be represented by the available samples.A 6 × 6 pixel chessboard reproduction illustrates how inadequate sampling prevents faithful representation of fine structure.
  • Aliasing: Image resolution imposes a limit on representable scale, so combinations of insufficient resolution and fine scale suffer from aliasing.Blurring moves an image toward coarser scales that can be represented without aliasing.
  • Object size: Images contain objects and structures at multiple sizes, but coarse scales can erase the high-frequency characteristics of smaller objects.The same image may include large objects such as people and animals alongside small objects such as hairs and branches.

B. Scale-variant and scale-invariant image representations

CNN training must account for task-relevant features that vary across resolution, object size, and image scale. The paper therefore combines scale-variant filters, which preserve scale-specific information, with a scale-invariant representation across scales.

  • Motivation: CNN filters must ignore task-irrelevant variation while capturing task-relevant features at their specific scales.The paper calls filters supporting the first behavior scale-invariant and those supporting the second scale-variant.
  • Why scale variance matters: Scale invariance alone is incomplete because reconstructing an image from scale-invariant components does not fully match the original image or natural-image statistics.This motivates retaining scale-variant image structure alongside invariant components.
  • Why scale variance matters: Recognition difficulty changes with image resolution because a very small object contains substantially less representational information than a much larger object.The cited example contrasts objects that are 3, 300, and 3000 pixels tall.
  • Proposed direction: The proposed multi-scale CNN explicitly handles resolution, object-size, and image-scale variation by learning scale-variant filters while constructing a scale-invariant representation.This is the paper’s central architectural proposal.

II. PREVIOUS WORK

Prior approaches often encourage scale invariance or expose CNNs to multiple scales, but they do not fully address scale-variant features across large scale differences. The paper proposes distributing analysis across specialized CNNs and combining their predictions.

  • Prior approaches: Explicit scale-invariance methods address invariant features but can forgo recognition of scale-variant features at multiple scales.This leaves a gap for tasks whose relevant image structure changes with scale.
  • Prior approaches: Scale jittering exposes CNNs to multiple scales during training as data augmentation, increasing training variation and potentially improving classification performance.Its benefits include more training data and reduced overfitting, alongside learning recognition across scales.
  • Prior approaches: Multi-scale training uses separate CNNs on resized image crops and averages their softmax posteriors at test time.This combines information from different scales and model averaging, improving performance over a single scale according to the cited prior work.
  • Paper contribution: The paper modifies multi-scale training to create multiple scales explicitly, reducing resizing aliasing and redundancy between scales.The approach is designed to support larger scale differences in the comparison.
  • Architecture motivation: A single CNN spanning fine-to-coarse characteristics would require many stages, greater complexity, more data, and more computation.The paper instead motivates distributing the task across specialized CNNs and combining their predictions.
  • Architecture motivation: The proposed representation associates a separate CNN with each Gaussian-pyramid scale, allowing scale-specific networks to learn both variant and invariant features.Higher pyramid levels are smoothed and down-sampled versions of preceding levels.

IV. IMAGE CLASSIFICATION TASK

The paper evaluates multi-scale CNNs on computational artist attribution, where fine details or coarse structures alone may be insufficient to distinguish artists. Combining information across scales is therefore suited to the task’s heterogeneous artwork sizes and resolutions.

  • Task motivation: Artist attribution asks which artist authored an artwork, and similar artists may be indistinguishable when only one spatial scale is considered.Fine details can miss scene-level or material differences, while coarse structure can miss material-specific details.
  • Task motivation: Successful artist attribution requires scale-variant features in addition to scale-invariant information.Different artists can resemble one another at fine scale but differ at coarse scale, or show the opposite pattern.
  • Computational assessment: Computational artwork classification has used both SIFT features and CNN representations for visual assessment.These approaches extend visual assessment beyond traditional expert practice to larger image datasets.
  • Computational assessment: Lower-resolution artwork datasets shift attention away from fine artist-specific details such as brushstrokes and materials toward coarser structures and shapes.A multi-scale approach can use information from both coarse and fine details.

V. EXPERIMENTAL SETUP

The experiment uses a fully convolutional single-scale CNN architecture and an ensemble of single-scale CNNs to produce multi-scale predictions. Training uses image crops, while testing operates on entire input images.

  • Architecture: The single-scale CNN is fully convolutional except for its final global average pooling layer.Strided convolutional layers replace conventional max or average pooling layers.
  • Analysis: Guided Backpropagation is used to visualize which parts of an input image are characteristic of a particular artist.The method computes gradients with respect to the input image during a backward pass.
  • Procedure: Training uses 224 × 224-pixel crops, whereas testing uses entire input images with shortest sides ranging from 256 to 2048 pixels.This setup is specified for the single-scale networks used in the experiment.
  • Architecture: The multi-scale CNN is an ensemble of single-scale CNNs whose softmax class posteriors are averaged for final predictions.The single-scale network is based on a previously established ImageNet architecture.

B. Dataset

The dataset contains digital reproductions of print artworks from the Rijksmuseum, with variation in physical size and pixel-to-millimeter ratios. Images are represented at four Gaussian-pyramid levels and divided into training, validation, and test sets.

  • Dataset: The dataset contains 58,630 digital reproductions of print artworks by 210 artists from the Rijksmuseum.The selection required paper printworks by a single artist, public-domain status, and at least 96 qualifying images per artist.
  • Scale variation: Print artworks retain variation in physical size and pixel-to-millimeter ratios, while resizing all images to one size confounds image resolution with physical resolution.The dataset is therefore suited to studying scale and resolution variation.
  • Scale variation: The study preserves scale and resolution variation because fixed pixel-to-millimeter normalization would lose visual detail.This choice supports learning both scale-invariant and scale-variant filters.
  • Image representation: A four-level Gaussian pyramid is created by resizing images to a 2048-pixel shortest side, then repeatedly smoothing and down-sampling by a factor of two.The resulting levels have shortest sides of 256, 512, 1024, and 2048 pixels.
  • Splits: The dataset is split into 70% training, 10% validation, and 20% test data.Reported results are based on the test set.

C. Training parameters

The networks use a stated learning rate, momentum, and weight decay, with learning-rate reductions triggered when validation error stops decreasing. The section also describes the test-set artwork dimensions.

  • Optimization: Training uses a learning rate of 10^-2, momentum of 0.9, and weight decay of 5 · 10^-4.These hyperparameters are used with the paper’s effective training procedure.
  • Artwork dimensions: The test-set artworks are predominantly square or slightly wider than tall, with most clustered around 250 × 250 mm.The scatter plot encodes artwork dimensions in millimeters and uses color to indicate local density.
  • Optimization: The learning rate is decreased by a factor of 10 whenever validation-set error stops decreasing.The reduction rule adapts the optimization schedule to validation performance.

D. Evaluation

Evaluation averages spatial predictions over entire images and reports Mean Class Accuracy, with additional comparisons across scales and against multi-scale training. Scale-wise correlations and high-dimensional prediction similarities assess whether scale-specific information contributes to classification.

  • Metrics: Mean Class Accuracy is the average accuracy across artists and is used because it is insensitive to unbalanced classes.It also enables comparison with results reported in prior work.
  • Evaluation procedure: The fully convolutional multi-scale CNN evaluates entire images without cropping by averaging each scale’s spatial output map.This produces one scale-specific prediction for each image.
  • Baselines: The study compares the proposed approach with multi-scale training using the same CNN architecture while varying only the input data.The comparison uses direct subsampling rather than Gaussian-blurred images before subsampling.
  • Scale analysis: Pairwise correlations between artists’ class accuracies at different scales indicate whether artist attribution is similar or different across scales.Low correlations suggest relevance of scale-specific information.
  • Reported results: The reported metrics include Mean Class Accuracy, mean recall, and F-score for individual scales and their four-scale ensemble.Additional Mean Class Accuracy results cover all possible scale combinations and multi-scale training.

VI. RESULTS

The multi-scale CNN performs best when combining information across scales, while artist attribution varies substantially by scale. Its representations capture similarities based on both fine and coarse artwork characteristics.

  • Overall performance: The four-scale ensemble outperforms every individual scale, and no combination of three or fewer scales surpasses it.The best-performing individual scale is 512.
  • Resolution effects: Performance decreases above resolution 512, suggesting a ceiling beyond which increasing resolution does not improve results.
  • Scale correlations: Adjacent scales have larger result correlations than non-adjacent scales, while multi-scale training produces generally stronger correlations.
  • Artist-specific variation: Artist attribution performance varies greatly across scales: some artists are highly scale-specific, whereas others remain stable across scales.The comparison uses standard deviation of mean class accuracy across scales.
  • Feature visualisation: Guided Backpropagation shows a shift toward finer details as resolution increases for the same artwork.
  • Artwork similarity: The t-SNE representation clusters visually similar artworks and expresses similarity through both fine and coarse characteristics, including similarities present at only some scales.

VII. DISCUSSION

The paper argues that an ensemble of scale-specific CNNs can represent both scale-variant and scale-invariant image features. It reports improved artist attribution, while noting computational and architectural trade-offs and unresolved questions about explicit scale-invariant learning.

  • Contribution: An ensemble of scale-specific CNNs learns features relevant at each scale, combining scale-variant and scale-invariant image structure.
  • Empirical findings: On artist attribution, the multi-scale CNN outperforms a single-scale CNN and state-of-the-art performance, with the best result obtained by combining all scales.
  • Trade-offs: The approach requires multiple scale-specific CNNs, whereas a single deeper CNN could be more complex, computationally costly, and data-intensive.
  • Comparison: Constructing a Gaussian pyramid increases performance and decreases correlations between scales compared with multi-scale training.
  • Trade-offs: The method uses more parameters than architectures with shared parameters, although the separate models can be trained independently and in parallel.
  • Open limitation: The approach does not explicitly learn scale-invariant features, which may emerge implicitly and motivates future work.
  • Scope: The authors expect multi-scale CNNs to help recognition tasks containing both fine- and coarse-grained task-relevant details, including scene, aerial, and biomedical image analysis.

VIII. CONCLUSION

The paper learns scale-variant and scale-invariant representations from high-resolution, multi-scale images using a multi-scale CNN. Its scale-specific representations are complementary, and combining them improves image-recognition performance.

  • Conclusion: The multi-scale CNN combines multiple single-scale CNNs to learn scale-variant representations across scales while encouraging scale-invariance.The approach analyzes multi-scale images containing both coarse and fine-grained details.
  • Conclusion: The approach improves performance on a computational artist attribution task and the current state of the art.
  • Conclusion: Representations learned at different scales differ in performance and image structure, yet remain complementary.
  • Conclusion: Averaging class posteriors across all scales produces optimal performance.
  • Conclusion: The authors conclude that jointly developing scale-invariant and scale-variant representations benefits recognition tasks involving image structure at varying scales and resolutions.

APPENDIX

The appendix table reports mean class accuracies for every scale combination under a specified mean-scale training procedure and marks the strongest combinations and overall score.

  • Appendix: The table reports mean class accuracies for all possible scale combinations.
  • Appendix: Results use the mean-scale training procedure described in [20], with '+' indicating that a scale is included.
  • Appendix: Bold identifies the best combined-performance combinations within each block, while underlining marks the best overall score.
Loading 1602.01255v2…