PaperlayerView exact revision on arXiv

Source-linked AI summary

Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models

Diego Marcos, Michele Volpi, Benjamin Kellenberger, Devis Tuia

arXiv:1803.06253v1cs.CV

TL;DR

Very-high-resolution remote-sensing images contain objects in arbitrary orientations, challenging CNNs that must learn redundant orientation-specific filters. The paper introduces RotEqNet, which rotates filters and propagates maximally activating orientations as vector fields; on two subdecimeter labeling benchmarks, it achieves strong performance with much smaller models.

  • Problem

    Very-high-resolution remote-sensing pixels often lack sufficient spectral information for confident labeling, while CNNs face arbitrary object orientations and substantial data and computational requirements.

  • Method

    RotEqNet rotates filters, retains the maximal activation and its orientation, and shares filter weights across angles to encode rotation equivariance.

  • Results

    On two subdecimeter land-cover labeling benchmarks, explicitly encoding rotation equivariance produced models between one and two orders of magnitude smaller than traditional CNNs and robust to varying training-data amounts.

  • Takeaways & Limitations

    Rotation-equivariant structure can support efficient, high-performing remote-sensing semantic labeling with limited labels and smaller models.

  • Takeaways & Limitations

    RotEqNet still requires rotated-filter interpolation, and computational time depends linearly on the number of rotations.

Abstract

from arXiv · show

In remote sensing images, the absolute orientation of objects is arbitrary. Depending on an object's orientation and on a sensor's flight path, objects of the same semantic class can be observed in different orientations in the same image. Equivariance to rotation, in this context understood as responding with a rotated semantic label map when subject to a rotation of the input image, is therefore a very desirable feature, in particular for high capacity models, such as Convolutional Neural Networks (CNNs). If rotation equivariance is encoded in the network, the model is confronted with a simpler task and does not need to learn specific (and redundant) weights to address rotated versions of the same object class. In this work we propose a CNN architecture called Rotation Equivariant Vector Field Network (RotEqNet) to encode rotation equivariance in the network itself. By using rotating convolutions as building blocks and passing only the the values corresponding to the maximally activating orientation throughout the network in the form of orientation encoding vector fields, RotEqNet treats rotated versions of the same object with the same filter bank and therefore achieves state-of-the-art performances even when using very small architectures trained from scratch. We test RotEqNet in two challenging sub-decimeter resolution semantic labeling problems, and show that we can perform better than a standard CNN while requiring one order of magnitude less parameters.

1. Introduction

Very-high-resolution remote-sensing labeling requires spatial reasoning while handling arbitrary object orientations. The paper proposes encoding rotation equivariance within CNNs to reduce model size and labeling requirements.

  • Semantic labeling assigns each pixel to a predefined land-cover or land-use class.
  • Very-high-resolution imagery sacrifices spectral resolution for spatial detail, so individual RGB or NIR pixels often lack sufficient information for confident classification.
  • CNNs avoid manual feature engineering but typically require extensive ground truth, difficult tuning, and substantial computational resources.
  • Because overhead-image orientation is generally arbitrary, semantic-label maps should rotate with rotated inputs, a property called equivariance.
  • Data augmentation can approximate rotation equivariance but requires independently learning similarly oriented filters and does not improve model compactness.
  • RotEqNet rotates each filter, retains the maximally activating orientation as a vector, and shares filter weights across angles, reducing required parameters by around one order of magnitude.

2. Rotation Equivariant Vector Field Networks (RotEqNet)

RotEqNet makes CNNs rotation equivariant by rotating filters and retaining only the maximal activation across orientations.

  • RotEqNet extends CNN architectures by rotating filters and selecting the maximal activation across rotations.

2.1. Convolutional neural networks for semantic labeling

CNNs build hierarchical spatial features through convolutional blocks and pooling, then combine multiscale activations for dense pixelwise semantic labeling.

  • CNN building blocks: CNNs cascade convolutions and nonlinear operations to transform an input image into a desired output, with depth determined by the number of operations.
  • CNN building blocks: A convolution applies the same filter over sliding local image patches to produce feature maps, whose dimensionality equals the number of filters.
  • CNN building blocks: Spatial max-pooling reduces activation-map size and lets later filters capture broader image context while supporting locally consistent and multiscale features.
  • CNN building blocks: Batch normalization normalizes feature maps within a batch, while dropout randomly removes feature maps during training to reduce dependence among filters.
  • From patch classification to (dense) semantic labeling: Dense semantic labeling predicts a class for every pixel rather than one label per image tile.
  • From patch classification to (dense) semantic labeling: Hypercolumns upsample activation maps to the original resolution, concatenate them with the image, and use their multiscale features for pixelwise classification.

2.2. From translation to rotation invariance

Standard CNNs obtain translation equivariance from shared local convolutions, whereas rotation equivariance requires representing responses across filter orientations. RotEqNet pools those responses into vector fields while preserving spatial structure.

  • From translation to rotation invariance: Standard convolutions reuse the same dot product over translated local neighborhoods, producing outputs that translate with the input aside from border effects.
  • From translation to rotation invariance: Rotation-aware convolutions compute responses to rotated copies of a canonical filter, encoding rotations as translations across an added orientation dimension.
  • From translation to rotation invariance: Orientation max-pooling retains the largest response and its angle at each location, preserving the image’s 2D arrangement while achieving rotation equivariance.
  • From translation to rotation invariance: The pooled output is a vector field carrying magnitude and orientation, and its memory requirements are independent of the number of rotations used.
  • From translation to rotation invariance: Max-pooling can discard useful information when multiple orientations are equally discriminant, but it reduces model and feature-map memory.

2.3. RotEqNet modules

RotEqNet builds rotation equivariance by convolving with rotated filters, pooling over orientations, and propagating activation magnitude and orientation as vector fields.

  • Rotating convolution: RotEqNet rotates each canonical filter across an orientation set spanning the full 360° for arbitrarily oriented remote-sensing objects.Other applications may restrict the angle range when orientations follow a known top-down relation.
  • Rotating convolution: Each rotating convolution produces R feature maps, one for each rotated filter, with a shared bias across rotations.Only the canonical filter is stored; rotated versions are obtained by resampling it with bilinear interpolation.
  • Orientation pooling: Orientation pooling retains the largest activation and its corresponding orientation at every spatial location, avoiding propagation of all R activations.The retained magnitude and angle form a polar representation of a two-dimensional vector field after nonnegative activation processing.
  • Dealing with vector inputs: The pooled polar representation is converted into Cartesian [u, v] components, preserving magnitude and orientation in a vector-field feature map.Because the components are orthogonal, vector-field convolution is computed as separate standard convolutions summed across components.
  • Dealing with vector inputs: Separate component convolutions enable deep architectures whose layers remain rotation equivariant.The architecture therefore accommodates vector-field inputs and outputs rather than relying solely on traditional scalar feature maps.

3. Data and setup

The experiments use two dense, sub-decimeter overhead-image benchmarks—Vaihingen and Zeebruges—with optical imagery, height information where available, and distinct class distributions.

  • 3.1.1. Vaihingen benchmark: The Vaihingen benchmark contains 33 orthorectified tiles at 9 cm resolution, with 16 fully annotated tiles available to participants.The imagery includes three channels, and the setup uses a normalized DSM whose heights are relative to the nearest ground pixel.
  • 3.1.1. Vaihingen benchmark: Vaihingen labeling covers six classes, with buildings and impervious surfaces comprising roughly half the data while cars and clutter each comprise about 2%.This produces substantial class imbalance in the benchmark.
  • 3.1.1. Vaihingen benchmark: The Vaihingen split uses 11 annotated tiles for training and five specified tiles for testing.The split follows the setup used in earlier benchmark studies.
  • 3.1.2. Zeebruges benchmark: Zeebruges consists of seven 10000 × 10000-pixel tiles at 5 cm resolution, with five labeled tiles for training and two for testing.The dataset provides RGB imagery and a LiDAR point cloud processed into a DSM.
  • 3.1.2. Zeebruges benchmark: Zeebruges uses eight classes, including water and boats, with water representing 30% of training data and cars plus boats only 1%.Harbour structures and cargo containers are largely labeled as clutter.

3.2. Experimental setup

The experiments compare compact RotEqNet models with larger standard CNN counterparts using matched architecture families, dense hypercolumns, augmentation, and training from scratch.

  • 3.2.1. CNN architecture: The RotEqNet hypercolumn architecture uses rotating convolutions before concatenation and standard 1 × 1 convolutions afterward.The caption states that all layers are rotation equivariant, including the 1 × 1 standard convolutions.
  • 3.2.1. CNN architecture: The baseline CNN keeps the same architecture and layer count but uses four times more filters per layer, yielding approximately 10 fold more parameters.Performance had started to saturate at this larger model size.
  • 3.2.1. CNN architecture: Each model has six convolutional layers with factor-of-two max-pooling downsampling and filter counts scaled by the single size parameter Nf.All convolutional filters are 7 × 7, followed by ReLU, batch normalization, upsampling, and concatenation.
  • 3.2.1. CNN architecture: Inputs are reshaped into 512 × 512 patches for Vaihingen and 500 × 500 patches for Zeebruges, while outputs remain arbitrarily dense prediction maps.The input size does not influence the reported results.
  • Training procedure: Training uses stochastic gradient descent with momentum 0.9, model-specific batch sizes, random 0°–360° rotations, flips, and improved Xavier initialization from scratch.Full rotation augmentation is retained for both models to make the comparison easier, although it is not strictly necessary for RotEqNet.

3.4. Vaihingen

The Vaihingen experiment varies the amount of available ground truth and compares a compact RotEqNet against a larger standard CNN with the same layer structure.

  • Ground-truth sensitivity: Three models per architecture use 4%, 12%, or 100% of the available Vaihingen training set.This design tests sensitivity to the amount of ground truth.
  • Model comparison: The comparison varies only filter count while keeping architecture structure and layer count fixed.RotEqNet is compared with a standard CNN of larger capacity but without built-in rotation equivariance.
  • Model sizing: Nf=3 for RotEqNet and Nf=12 for the standard CNN were selected as the smallest models exceeding 87% overall accuracy on validation data.The resulting models used approximately 10^5 and 10^6 parameters, respectively.

3.5. Zeebruges

For the more complex Zeebruges dataset, the study increased RotEqNet capacity and trained three model sizes with different filter counts.

  • 3.5. Zeebruges: RotEqNet models used Nf = [4, 5, 7] to address the greater complexity of Zeebruges.Training lasted 34 epochs with staged learning rates of 1·10^-2, 2·10^-3, and 4·10^-4.

4. Results and discussion

RotEqNet achieves strong semantic-labeling performance on Vaihingen and Zeebruges while using substantially fewer parameters, with rotation-aware features improving detection of small, oriented objects. Its accuracy and efficiency depend on the number of orientations and can involve less-smooth boundaries or increased computational cost.

  • Vaihingen results: Over 87% OA is achieved by both models on the full Vaihingen dataset, while using 4% of training data reduces performance to around 84.7%.RotEqNet reaches 87.6% OA on the benchmark server’s withheld test set.
  • Vaihingen results: RotEqNet’s advantage is more apparent in AA, largely because of improved car detection, and its performance gap over the standard CNN increases with more ground truth.The authors suggest oriented filters help detect edge-like features and let the model focus on semantic labeling rather than learning rotation invariance.
  • Learned filters and mechanism: RotEqNet avoids separately learning rotated filter copies by applying filters at multiple orientations and retaining the largest activations.Its first-layer filters use a disk-shaped support within the 7 × 7 grid, and their canonical orientation is arbitrary.
  • Qualitative comparison: RotEqNet performs better on smaller objects such as cars and grass paths, but its classification maps contain less-smooth edges.The authors associate the rougher edges possibly with neighboring pixels selecting different feature orientations.
  • Zeebruges results: 1.4·10^5 parameters enables RotEqNet to achieve comparable Zeebruges results to CNN models containing on the order of 10^7 parameters.The comparison includes VGG/SVM with 2.5·10^7 parameters and AlexNet with around 6·10^7 parameters.
  • Computational efficiency: With R = 16, RotEqNet outperforms the standard CNN in forward-pass speed up to R = 64, although interpolation and orientation-dependent convolutions can increase computational time.Reducing the number of feature maps can compensate for this cost when the number of orientations is sufficiently small.

5. Conclusion

RotEqNet encodes rotation equivariance through rotating convolutions and maximal-activation propagation, reducing model requirements while retaining orientation information. Across two subdecimeter benchmarks, it produced much smaller models and robustness to varying training-set sizes, with potential beyond semantic labeling.

  • 5. Conclusion: RotEqNet applies rotating convolutions and propagates maximal activations to encode rotation equivariance while reducing model size and runtime memory requirements.The propagated maxima retain most orientation information while avoiding full orientation feature maps.
  • 5. Conclusion: One to two orders of magnitude smaller models were achieved than traditional CNNs on two subdecimeter land cover semantic labeling benchmarks.
  • 5. Conclusion: Structural rotation equivariance, rather than data augmentation, provided robustness to varying amounts of training data.This supports efficient training when available labels vary substantially across remote sensing acquisition modes and problems.
  • 5. Conclusion: RotEqNet’s logic can extend to convolutional deep models for detection and regression tasks requiring rotation invariance, equivariance, or covariance.Examples include detecting cars, airplanes, and trees, or estimating superresolution and biophysical parameters when labeled instances are limited.
Loading 1803.06253v1…