Multi-Granularity Canonical Appearance Pooling for Remote Sensing Scene Classification

I. INTRODUCTION

• Remote-sensing scene classification is challenged by missing alignment between visual features and semantic labels, as well as intra-class diversity and inter-class similarity.The paper notes that visually similar scenes can have different labels, while images in the same category can look different.Source: §I · p. 1
• Detailed annotations are impractical to collect because they require substantial manpower and are time-consuming and subjective, especially for hierarchical categories.The introduction describes superordinate, basic, and subordinate category levels in datasets such as NWPU-RESISC45 and AID.Source: §I · pp. 1–2
• MG-CAP learns granular representations to incorporate latent ontological structure and alleviate visual-semantic discrepancies without relying on manually designed ontologies.The framework progressively crops images, pools canonical appearances across transformations, and uses a Siamese architecture to learn transformation-invariant features.Source: §I · p. 2

A. Handcrafted Feature-based Methods

• Handcrafted local features capture regional shape and structure, but typically require conversion into intermediate descriptors for whole-image representation.Examples include SIFT and HOG.Source: A. Handcrafted Feature-based Methods · p. 2
• Handcrafted global and texture descriptors represent image colour, spatial structure, or texture, but some colour methods are sensitive to illumination changes or quantisation errors.Examples include ACC, BIC, LCH, GIST, Gabor, GLCM, LBP, and LAS.Source: A. Handcrafted Feature-based Methods · p. 2

B. Mid-level Feature Learning-based Methods

• Mid-level methods include PCA, codebook generation with K-means, sparse coding, and autoencoders that compress features by minimizing reconstruction error.The paper frames unsupervised learning as reducing reliance on labour-intensive labels and removing low-level feature outliers.Source: B. Mid-level Feature Learning-based… · pp. 2–3

C. Deep Neural Network based Methods

• Deep-learning work for remote-sensing classification includes CNN feature fusion, transfer learning, hierarchical features, metric learning, weak supervision, attention, and transformation- or view-invariant representations.The related-work discussion also notes second-order methods such as bilinear pooling, which can suffer from high dimensionality and visual burstiness.Source: C. Deep Neural Network based Methods · p. 3

A. Overview

• MG-CAP progressively crops images into multiple granularities, transforms each region, and uses Siamese CNN features to build Gaussian covariance matrices for canonical-appearance pooling.It fuses granularity-specific SPD matrices by channel-wise averaging, then applies EIG-based matrix normalisation to improve feature discriminability.Source: A. Overview · pp. 3–4

B. Canonical Appearance Pooling Layers

• Canonical appearance pooling addresses nuisance transformations associated with large within-class variation, while multiple-instance averaging does not capture dependencies among instances.The paper describes canonical appearance pooling as learning transformation-invariant features.Source: B. Canonical Appearance Pooling Layers · p. 4
• At each granularity, a max operator selects the optimal feature representation from transformed instances, producing a transformation-invariant feature.The transformation set considered in this paper consists of rotations.Source: B. Canonical Appearance Pooling Layers · p. 4
• A Siamese-style architecture shares weights across transformed instances to capture their dependencies, while neighboring granularities retain independent weights.The architecture is deployed in a multi-stream fashion.Source: B. Canonical Appearance Pooling Layers · p. 4
• Covariance matrices encode second-order statistics of local CNN features and preserve spatial structure that vectorized first-order features may lose.Gaussian covariance matrices additionally incorporate first-order and second-order information.Source: B. Canonical Appearance Pooling Layers · pp. 4–5
• A logarithmic operation flattens the SPD manifold for Euclidean distance measurements, and a small ridge regularizes the Gaussian covariance matrix.The ridge converts a symmetric positive semi-definite matrix into a symmetric positive definite matrix.Source: B. Canonical Appearance Pooling Layers · p. 5
• Stacking and averaging SPD matrices fuses multi-grained information while avoiding the feature-dimension expansion caused by concatenation.The approach preserves the dimensionality of the feature representation G+.Source: B. Canonical Appearance Pooling Layers · p. 5

C. EIG-decomposition Layers

• Log-E maps the SPD manifold to a Lie-group structure for Euclidean operations, while Sqrt-E is presented as a more numerically stable alternative in nonlinear settings.The paper notes that Log-E can be numerically unstable in nonlinear scenarios.Source: C. EIG-decomposition Layers · p. 5
• Log-E requires strictly positive eigenvalues and can overstretch small eigenvalues or reverse their order, motivating eigenvalue rectification.The rectification function is introduced in.Source: C. EIG-decomposition Layers · p. 5
• The rectification function floors diagonal eigenvalues at ε to keep normalized eigenvalues positive and improve numerical stability for Log-E.ε is a threshold and I is the identity matrix.Source: C. EIG-decomposition Layers · p. 5
• A proxy parameter R(i, i) lets the rectification approach call the EIG-decomposition function only once, reducing computational demands.The comparison is with an approach that incorporates rectification into an extra EIG-decomposition function.Source: C. EIG-decomposition Layers · p. 6

D. Back-propagation

• When EIG input matrices are degenerate, the gradient can approach infinity; replacing infinite values with zero prevents back-propagation from being interrupted.Degeneracy may occur when normalized eigenvalues are identical, leaving corresponding eigenvectors arbitrary.Source: D. Back-propagation · p. 6
• The derivation uses a chain rule with first-order Taylor expansion and approximation, and provides gradients for normalization and rectification as well as standard EIG decomposition.It also derives gradients for EIG, SPD-matrix, and canonical-appearance-pooling layers.Source: D. Back-propagation · pp. 6–7
• Gradients for the integrated framework are computed by cascading the layer gradients across granularities because MG-CAP is fully differentiable.The derivation presents the EIG, SPD-matrix, and canonical-appearance-pooling layers in sequence.Source: D. Back-propagation · p. 7

A. Experimental Datasets

• Table I summarizes statistics for the datasets used in the paper.Source: Table I · p. 7
• The experiments use NWPU-RESISC45, AID, and UC Merced for remote-sensing scene classification.Source: A. Experimental Datasets · p. 7
• Spatial resolutions are 0.2m to 30m for NWPU-RESISC45, 0.5m to 8m for AID, and 0.3m for UC Merced.Source: A. Experimental Datasets · p. 7

B. Implementation Details

• VGG-D pretrained on ImageNet supplies features through conv5_3, with random 224 × 224 crops and horizontal flips used for augmentation.Transformed images are zero-padded to 317 × 317 and resized to 224 × 224 before entering the Siamese architecture.Source: B. Implementation Details · p. 7
• Training first fits the classification layer at learning rate 0.1, then fine-tunes the full network at 10^-3 with learning-rate decay.The experiments use momentum 0.9, weight decay 5 × 10^-4, and batch size 12 for three granularities and 12 transformations.Source: B. Implementation Details · p. 7
• Results are averaged over ten random training/testing splits and reported with standard deviations.The authors set λ to 1 × 10^-4 and use ε to clip eigenvalues into [1 × 10^-5, 1 × 10^5] for both Sqrt-E and Log-E.Source: B. Implementation Details · p. 7

C. Experimental Results and Comparisons

• Deep-learning methods outperform the listed handcrafted and unsupervised feature-learning methods, while VGG-D plus BoVW performs best among registered methods in the cited comparison.Transferred neural networks with a linear SVM reach about 76% at the 10% training split, with about a further 3% at 20%.Source: C. Experimental Results and Comparisons · p. 8, Table II · p. 8
• 90.64% and 92.75% accuracy make Sqrt-E MG-CAP the reported best on NWPU-RESISC45 across the two training cases.The comparisons use three granularities and 12 rotations per granularity; the table reports accuracy for MG-CAP, deep-learning baselines, and state-of-the-art methods.Source: C. Experimental Results and Comparisons · p. 8, Table III · p. 9
• On AID, Sqrt-E MG-CAP reaches 93.34% at a 20% training ratio, exceeding DCNN by 2.52% and RTN by 1.1%.At a 50% training ratio, DCNN is slightly better; with VGG-D, MG-CAP is below DCNN, MSCP, and RTN.Source: C. Experimental Results and Comparisons · p. 8, Table III · p. 9
• 99.0% accuracy makes Sqrt-E MG-CAP the best among compared methods on UC-Merced.The passage notes that UC-Merced has 21 categories and that listed deep-learning methods achieve similar results with many training samples.Source: C. Experimental Results and Comparisons · pp. 8–9
• 43 categories exceed 80% accuracy and 31 exceed 90%; Church and Palace improve by 8% and 13%, respectively, over results reported in.The passage presents these as category-level results in Fig. 3.Source: Figure 3 · p. 8, C. Experimental Results and Comparisons · p. 9

• Combining three granularities achieves the best results among the six tested cases.Source: 1) Effect of Granularity · p. 9
• 87.13% versus 86.57% with Log-E and 90.17% versus 89.82% with Sqrt-E favor combining the first and second over the second and third granularities.The authors suggest finer crops can improve accuracy, but over-cropping may damage performance.Source: 1) Effect of Granularity · p. 9
• The second granularity is the most accurate individual level, while the third is the least accurate.Source: 1) Effect of Granularity · p. 9

2) Influence of Transformations:

• About 3% accuracy improvement comes from rotating three times per granularity; gains then stabilize, while more transformations raise memory burden, so experiments use 12.Source: 2) Influence of Transformations · p. 9

E. Qualitative Visualisation and Analysis

• Visually similar images tend to share canonical orientations, while Grad-CAM heatmaps vary with image content; a partially present Palace object changes its canonical appearance.Fig. 5 uses blue, yellow, and green dashed lines for the first, second, and third granularities, respectively.Source: E. Qualitative Visualisation and… · pp. 9–10, Figure 5 · p. 10
• MG-CAP requires 55.99 MB, while RTN takes an extra 12.7 MB; RTN has shorter inference time, and GPU MG-CAP predicts in 0.216 seconds.Source: E. Qualitative Visualisation and… · p. 10, Table V · p. 10

V. CONCLUSION

• MG-CAP learns latent dataset ontologies through multi-granularity cropping, maxout-based Siamese feature extraction, Gaussian covariance matrices, matrix normalisation, and GPU-supported EIG decomposition.The framework is trained end to end; future work will investigate compressing model parameters while maintaining classification performance.Source: §V · p. 10