PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Scale-spaces: Equivariance Over Scale

Daniel E. Worrall, Max Welling

arXiv:1905.11697v1cs.LGcs.CVstat.ML

TL;DR

CNNs do not explicitly fuse information across scales, especially for non-invertible transformations such as downscaling. The paper introduces semigroup-based scale-equivariant correlations and integrates them into deep networks, finding mostly equivariant activations and evaluating the models on Patch Camelyon and Cityscapes.

  • Problem

    Existing scale-handling methods lack explicit mechanisms to fuse information from different scales, while downscaling is a non-invertible transformation that deletes information.

  • Method

    The paper extends convolutions through semigroup-equivariant correlations and uses scale-translation actions to build deep scale-space networks.

  • Results

    The average equivariance error is below 0.01 for three DSSs evaluated with random weights and an eight-scale truncated scale-space.

  • Takeaways & Limitations

    Deep scale-spaces generalize convolutional neural networks and provide scale-equivariant operations usable in modern neural architectures.

  • Takeaways & Limitations

    Finite scale-space truncation breaks global equivariance, while adding a scale dimension increases computational overhead and correlation complexity.

Abstract

from arXiv · show

We introduce deep scale-spaces (DSS), a generalization of convolutional neural networks, exploiting the scale symmetry structure of conventional image recognition tasks. Put plainly, the class of an image is invariant to the scale at which it is viewed. We construct scale equivariant cross-correlations based on a principled extension of convolutions, grounded in the theory of scale-spaces and semigroups. As a very basic operation, these cross-correlations can be used in almost any modern deep learning architecture in a plug-and-play manner. We demonstrate our networks on the Patch Camelyon and Cityscapes datasets, to prove their utility and perform introspective studies to further understand their properties.

1 Introduction

Scale is fundamental to visual data, but CNNs are predominantly local and existing scale-handling methods do not explicitly fuse information across scales. The paper addresses this by extending convolutions to non-invertible scale transformations.

  • 1 Introduction: CNNs are predominantly local because they use small filters, leaving multiscale interactions in deeper layers insufficiently understood.Empirical work with dilated convolutions suggests that multiscale reasoning remains incomplete.
  • 1 Introduction: Scale averaging, scale selection, and scale augmentation help handle scale but lack explicit mechanisms to fuse different scales into one representation.The proposed generalized convolution instead takes information from different scales as input.
  • 1 Introduction: Naïve ×1/8 subsampling destroys image structure, whereas bandlimiting before subsampling preserves much of the low-frequency structure.The figure contrasts direct subsampling with blur-based preprocessing before reduction.
  • 1 Introduction: Many transformations, including image downscaling, are non-invertible because they delete information that cannot be recovered.The paper treats these transformations as symmetry operations that require semigroup rather than group structure.
  • 1 Introduction: The paper introduces semigroup-equivariant correlations and a scale-equivariant CNN to extend convolutions to non-invertible transformations.The construction is motivated by scale transformations such as downscaling.

2 Background

The background develops scale-spaces, semigroups, actions, lifting, and equivariant correlations as the mathematical framework for modeling image downscaling. Scale-space representations progressively blur images, and their semigroup structure captures irreversible information loss.

  • 2 Background: Downsampling should blur an image before subsampling; otherwise aliasing introduces frequencies that the reduced resolution cannot represent.The paper models downsampling as convolution with a blur kernel followed by sampling on a coarser grid.
  • 2 Background: Scale-spaces represent an image together with multiple blurred versions, commonly using Gaussian convolution.Higher scale levels remove more high-frequency detail.
  • 2 Background: Gaussian scale-spaces satisfy a semigroup property: blurrier levels can generate other levels, but sharper images cannot be recovered by Gaussian convolution.This one-way progression encodes information loss.
  • 2 Background: Semigroups generalize groups by requiring associative composition without necessarily providing identity elements or inverses.Their elements can represent dilation parameters and transformations that are not invertible.
  • 2 Background: A semigroup action maps domain elements through transformations whose composition follows the semigroup structure, with left actions applying composed factors in reverse written order.The action is closed under composition and can also operate on functions.
  • 2 Background: The scale-translation semigroup uses dilation and shift tuples, with an anisotropic Gaussian action that bandlimits before dilating and shifting the image.This action maps functions on the discrete image domain back to that domain for fixed transformation parameters.
  • 2 Background: Lifting maps a function on the original domain to a function on the semigroup, where complicated transformations become shifts in the lifted representation.For lifted functions, the action is expressed as f ↑(s) 7→ f ↑(st).
  • 2 Background: Group correlations compute inner products across transformations and are equivariant because the transformation action commutes with correlation.The output is defined on the transformation domain.

3 Method

The method extends correlation from groups to semigroups, enabling scale-equivariant processing under scale-space actions. It lifts signals to a discretized scale-space, uses transformed signals in the correlation, and incorporates compatible nonlinearities and finite-scale implementation choices.

  • Semigroup correlation: Semigroup correlation extends convolutional correlation to transformations that may be non-invertible, including scale changes.Unlike group correlation, it transforms the signal rather than the filter because inverse-based variable changes are unavailable in general semigroups.
  • Semigroup correlation: The semigroup correlation computes filter–signal inner products under semigroup actions and produces responses indexed by the semigroup.This construction is equivariant to left actions on the signal and can recover standard convolution or group correlation as special cases.
  • Scale-space correlation: The scale-space correlation chooses the scale-space action and applies it to signals lifted over a discrete set of dilation and translation parameters.The dilation discretization uses a countable semigroup, with lifted signals and filters indexed by scale and spatial coordinates.
  • Scale-space correlation: For the experimental form, scaling a filter corresponds to a dilated convolution between ψℓ and the scale-space slice fℓ+k.The schematic shows dilation paired with a shift to a corresponding level in the scale-space.
  • Network operations: Pointwise nonlinearities commute with the lifted-function action, permitting standard pointwise nonlinearities in the equivariant network.Batch normalization computes statistics over all activation dimensions except channels.
  • Implementation and boundaries: Finite scale-space truncation breaks global equivariance, so the implementation interleaves scale-dimension 2 and 1 filters to balance multiscale interactions against boundary effects.The scale-space uses four layers, integer powers-of-two dilations, maximum dilation 8, and kernel width 33.

4 Experiments and Results

The experiments evaluate DSS networks on Patch Camelyon and Cityscapes, alongside empirical tests of scale equivariance. Results show improved task performance over a non-equivariant baseline and generally low equivariance error, except near truncation boundaries.

  • Task performance: DSS models are evaluated on Patch Camelyon classification and Cityscapes semantic segmentation, including comparisons with non-equivariant baselines.Patch Camelyon uses a 4-scale S-DenseNet; Cityscapes uses a 4-scale S-ResNet with variants including multiscale interaction layers.
  • Task performance: Both scale- and rotation-equivariant models outperform the non-equivariant DenseNet baseline on Patch Camelyon accuracy.
  • Equivariance quality: Equivariance quality is measured as a normalized L2-distance between activations of a dilated image and the theoretical action on activations.
  • Equivariance quality: Average equivariance error is below 0.01 for three randomly weighted DSSs with scale-space truncated to 8 scales.The reported errors arise from truncation of the discrete Gaussian kernels used to lift the input to scale-space.
  • Equivariance quality: Equivariance errors increase when receptive fields overlap the scale-space truncation boundary.Figure 4 compares 1-, 2-, and 3-layer DSSs, with each line representing the error defined in Equation 25.

5 Related Work

Related work extends equivariant modeling to rotations, reflections, and other transformation groups, while scale-focused approaches remain less numerous. Existing scale methods include input reparameterization and filter scaling, but have stated limitations.

  • Group convolutions: Prior group-convolution work addresses continuous and discrete roto-translations, spherical rotations, and in-plane reflections in two and three dimensions.
  • Scale methods: Scale-focused methods include log-polar reparameterization before standard CNNs and group-CNN architectures that scale filters instead of rotating them.
  • Scale methods: Log-polar methods are sensitive to the predefined origin, while filter-scaling approaches ignore large scale variation.

6 Discussion, Limitations, and Future Works

The discussion identifies boundary effects and computational overhead as important limitations of DSS, while proposing broader semigroup applications and future work across scales and domains.

  • Limitations and mitigation: Best-performing architectures mainly use scale-dimension-one correlations interleaved with higher scale-dimension correlations, which the authors associate with boundary effects.
  • Future work: Smaller jumps between scale-space levels may mitigate boundary effects, but non-integer dilations would require interpolation.The proposed dilation form is A_k = c^-kI for 1 < c < 2.
  • Limitations and mitigation: Adding a scale dimension to activations increases computational overhead, and correlation complexity grows exponentially with the number of model symmetries.
  • Future work: Future work includes semigroup-structured transformations such as causal shifts, occlusions, and affine transformations.
  • Future work: The authors also propose exploring multiscale interactions on meshes and graphs, where symmetry is less well-defined.

7 Conclusion

The paper presents deep scale-spaces as a CNN generalization that exploits scale symmetry. It develops semigroup correlations, relates them to standard and group convolutions, and applies them to scale-translation actions in neural architectures.

  • Conclusion: Deep scale-spaces generalize convolutional neural networks by exploiting scale symmetry in conventional image-recognition tasks.
  • Conclusion: The paper develops a semigroup correlation as a generalized convolution operator and derives standard and group convolutions from it.
  • Conclusion: The semigroup correlation is tailored to the scale-translation action used in classical scale-space theory and incorporated into modern neural architectures.

A Scale-spaces

This section directs readers to additional literature on scale-spaces and lists several foundational references.

  • The section provides extra information on scale-spaces for interested readers.It recommends consulting the cited literature for an in-depth treatment.
  • The suggested literature includes works by Florack, Pauwels, Lindeberg, Crowley, Salden, Duits, and Burgeth.
  • The references span publications from 1990 through 2007.

A.1 1D Gaussian Scale-space

The 1D construction models downscaling through an action on a bandlimited signal, then corrects the resulting bandlimit shift with Gaussian smoothing.

  • 1D Gaussian Scale-space: The construction starts with a 1D signal f0 having intrinsic bandlimit s0 and downsizes it by dilation a using La[f](x) = f(a^-1x).The dilation satisfies a ≤ 1.
  • 1D Gaussian Scale-space: Downscaling by a shifts the bandlimit from s0 to a^2s0, moving the signal’s frequency content higher.The passage attributes this shift to the narrower blurring Gaussian after downscaling.
  • 1D Gaussian Scale-space: To preserve the original bandlimit on a discrete grid, the downscaled signal is convolved with a correcting Gaussian of width s0 − a^2s0.This correction restores the maximum representable frequency content.
  • 1D Gaussian Scale-space: Equation (29) expresses how Gaussian convolution and dilation can be reordered while transforming the Gaussian scale parameter.
  • 1D Gaussian Scale-space: The section then imposes the requirement that the downscaled signal retain bandlimit s0.
  • 1D Gaussian Scale-space: A 1D Gaussian scale-space parameterized by dilation is constructed by setting the scale-space according to this corrected downscaling relation.

A.2 ND Gaussian Scale-space

The multidimensional construction extends Gaussian scale-spaces to affine image transformations while preserving a fixed bandlimit, and situates this model among alternative scale-spaces.

  • ND Gaussian Scale-space: The multidimensional treatment represents an initial image f0 and extends the downscaling analysis beyond one dimension.
  • ND Gaussian Scale-space: A Gaussian represents the initial image’s intrinsic bandlimit, and an affine action applies an affine transformation to the signal.
  • ND Gaussian Scale-space: Resizing a bandlimited signal transforms its bandlimit as Σ0 7→ AΣ0A⊤.
  • ND Gaussian Scale-space: A second Gaussian convolution with width Σ0 − AΣ0A⊤ restores the original bandlimit before and after resizing.The construction exchanges Gaussian convolution and affine action while changing the bandwidth representation.
  • ND Gaussian Scale-space: The affine action requires the Gaussian correction covariance to be symmetric and positive definite, which constrains the admissible transformations.
  • ND Gaussian Scale-space: The singular values of A may not exceed unity, so rotations remain allowed while upscalings are prohibited.
  • Alternative scale-spaces: The section also introduces α-scale-spaces, discrete Gaussian scale-spaces, and binomial scale-spaces as alternative constructions.The α-scale-space returns to the standard Gaussian scale-space when α = 1; binomial filters obey a semigroup property and converge toward Gaussian kernels as N grows.
  • Alternative scale-spaces: Binomial scale-spaces use discrete convolution kernels whose variance approaches σ^2 = N/4 as N grows.

B Architectures In The Experiments

The experiments instantiate scale-equivariant correlations within DenseNet and ResNet architectures, using discrete Gaussian scale-spaces and specified transition, residual, and scale-channel configurations.

  • Experimental setup: The experimental models use DenseNet and ResNet backbones with four scales of a discrete Gaussian scale-space.
  • ResNet: A residual block combines two scale-equivariant correlations with batch normalization and ReLU nonlinearities.
  • ResNet: The residual sequence is corr[1, 3, 3] - BN - ReLU - corr[k, 3, 3] - BN.
  • ResNet: In corr[k, h, w], k is the number of scale channels, while h and w specify the spatial filter height and width.
  • ResNet: The ResNet model uses bilinear upsampling at the network’s end because segmentations are described as lacking high-frequency details.
  • DenseNet: DenseNet consists of three dense blocks, with each layer concatenating all previous layer outputs.
  • DenseNet: Each DenseNet layer applies batch normalization, ReLU, and corr[k, 3, 3] in sequence.
  • DenseNet: Between dense blocks, transition layers halve channels with a 1 × 1 convolution and then apply spatial average pooling with kernel size 2 and stride 2.
Loading 1905.11697v1…