PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Transformer Networks for Compact Spherical Convolution

Yu-Chuan Su, Kristen Grauman

arXiv:1812.03115v2cs.CV

TL;DR

Existing ways to transfer perspective-trained CNNs to 360° images incur substantial computation or accuracy losses. KTN transforms source kernels for equirectangular imagery, and the paper reports that it outperforms existing methods while offering scalability and transferability. The same KTN can serve source CNNs of a shared architecture across tasks without retraining.

  • Problem

    Existing transfers of perspective-trained CNNs to 360° images are computationally expensive when repeatedly projecting images or inaccurate when applied directly to distorted equirectangular images.

    Source: §1 · p. 1

  • Method

    KTN learns a function that transforms source CNN kernels for equirectangular 360° images, accounting for distortion by polar angle and kernel.

    Source: §1 · p. 2

  • Results

    KTN outperforms existing spherical-convolution methods while providing superior scalability and transferability.

    Source: §5 · p. 8

  • Takeaways & Limitations

    One KTN can be applied without additional training to source CNNs with the same base architecture but different target tasks.

    Source: §3.4 · p. 5

    The method is developed for equirectangular 360° images, where distortion varies by row and kernel receptive fields may vary across rows.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Ideally, 360{\deg} imagery could inherit the deep convolutional neural networks (CNNs) already trained with great success on perspective projection images. However, existing methods to transfer CNNs from perspective to spherical images introduce significant computational costs and/or degradations in accuracy. In this work, we present the Kernel Transformer Network (KTN). KTNs efficiently transfer convolution kernels from perspective images to the equirectangular projection of 360{\deg} images. Given a source CNN for perspective images as input, the KTN produces a function parameterized by a polar angle and kernel as output. Given a novel 360{\deg} image, that function in turn can compute convolutions for arbitrary layers and kernels as would the source CNN on the corresponding tangent plane projections. Distinct from all existing methods, KTNs allow model transfer: the same model can be applied to different source CNNs with the same base architecture. This enables application to multiple recognition tasks without re-training the KTN. Validating our approach with multiple source CNNs and datasets, we show that KTNs improve the state of the art for spherical convolution. KTNs successfully preserve the source CNN's accuracy, while offering transferability, scalability to typical image resolutions, and, in many cases, a substantially lower memory footprint.

1. Introduction

The introduction frames spherical recognition as a transfer problem: existing approaches are either computationally expensive or inaccurate, while spherical-specific models bring other trade-offs. KTN transforms source CNN kernels for 360° images and is presented as transferable and compact while outperforming existing methods.

1. Introduction

  • Repeated tangent-plane projections are expensive, while applying a CNN directly to the equirectangular image is inaccurate because its visual content is distorted.Other spherical-specific approaches also trade accuracy, model size, memory, or task-specific retraining against one another.Source: §1 · pp. 1–2
  • KTN learns a function that transforms source CNN kernels for equirectangular 360° images, accounting for distortion using the polar angle and source kernel.It is trained to reproduce source-CNN outputs on tangent-plane projections while avoiding repeated image projection.Source: §1 · p. 2
  • A KTN trained for a base architecture can transfer multiple source CNNs, including a VGG model from ImageNet classification to Pascal object detection.The source CNN is an input to the KTN rather than having its kernels embedded in the KTN parameters; the VGG example reuses the same KTN.Source: §1 · p. 2
  • KTN is orders of magnitude smaller than SphConv and is more data efficient and accurate than Spherical U-Net and SphereNet.It requires no annotated 360° training images, and the cited accuracy comparison is attributed to avoiding their feature interpolation assumption.Source: §1 · p. 2

2. Related Work

Related work adapts conventional CNNs to spherical imagery through projection, spherical-specific kernels, or spectral convolution, each with trade-offs. KTN is distinguished by learning a transformation that accounts for spatial and cross-channel correlations and transfers across tasks for a shared architecture.

2. Related Work

  • Cubemap-trained vanilla CNNs reduce distortion relative to equirectangular input, but cubemap distortion and discontinuities remain, with sub-optimal accuracy on tasks such as object detection.The passage describes cubemap approaches as improving accuracy over earlier direct equirectangular methods.Source: §2 · p. 2
  • SphConv learns separate kernels by equirectangular row, while SphereNet and Spherical U-Net share weights but rely on feature interpolation in the equirectangular plane.The passage says SphConv is more accurate than a vanilla CNN but significantly increases model size; the interpolation assumption of the other two is described as problematic.Source: §2 · p. 2
  • KTN learns transformations that consider spatial and cross-channel correlation, and the same model can transfer across source CNNs with the same architecture without retraining for new tasks.The passage contrasts this with prior methods that require training a new model for each task.Source: §2 · p. 2
  • Spectral spherical CNNs [15] can require substantial memory for cached basis functions and frequency-domain feature maps, limiting viable input resolution.The passage also notes that removing orientation can unnecessarily restrict discrimination when orientation is semantically significant.Source: §2 · p. 2
  • Spatial transformer networks, active convolution [21], and deformable convolution [13] model geometric transformations, but not the location-dependent transformations and cross-channel correlations described for 360° images.The related-work passage contrasts these methods with KTN's handling of spherical-image properties and feature-channel correlation.Source: §2 · pp. 2–3

3. Approach

KTN transfers perspective-CNN convolutions to equirectangular 360° images by transforming kernels according to spherical distortion and training those transformations to reproduce source-CNN features. Its row-dependent architecture accommodates changing receptive fields, while its function-based design supports model transfer across tasks that share a base architecture.

3.1. KTN for Spherical Convolution

  • KTN replaces shared-kernel convolution with a transformation that generates a kernel for the distortion at each location on the sphere.The transformed kernel is applied to the equirectangular feature map, and the distortion parameter Ω varies over the sphere.Source: §3.1 · p. 3

3.2. KTN Architecture

  • In equirectangular projection, distortion depends on polar angle, which maps to image row and lets one transformed kernel serve an entire row.This row mapping supports more efficient computation than generating a separate kernel for every location.Source: §3.2 · p. 3
  • KTN uses row-dependent channel-wise projections to resize source kernels to each row’s receptive field, followed by depthwise-separable convolution blocks.This architecture addresses the variable kernel sizes needed as receptive fields change with polar angle.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. KTN Objective and Training Process

  • Training matches KTN feature maps to source-CNN outputs computed on tangent-plane projections, optimizing across the full feature map rather than one polar angle.The objective uses arbitrary 360° images and does not require annotated target-task data.Source: §3.3 · p. 4

3.4. Discussion

  • The same KTN can transfer different task-specific source CNNs without additional training when they share a base architecture, and it preserves orientation-specific features.The source kernels are inputs to KTN rather than parameters embedded in the KTN model.Source: §3.4 · pp. 5–6
  • For five VGG-based CNNs, SphConv requires 29×5=145GB of space, while KTN takes 56×5+14=294MB; the methods perform similarly in accuracy.SphConv has hundreds of times more parameters than KTN, and requires a separate model for each source CNN.Source: §3.4 · p. 5
  • SphereNet’s feature interpolation is only approximate beyond the first layer because nonlinear activations break equivalence; cube-padding can degrade accuracy, while dense tangent-plane sampling is computationally costly.The paper attributes the cube-padding issue to objects taking different appearances on tangent planes, especially with large fields of view.Source: §3.4 · pp. 5–6

4. Experiments

The experiments evaluate KTN’s accuracy, transferability across source models, and memory and computation overhead. KTN matches strong accuracy baselines, transfers across same-architecture models, and combines compactness with improved accuracy in the reported comparisons.

4. Experiments

  • The evaluation covers Spherical MNIST, Pano2Vid, and Pascal VOC, using classification accuracy, convolution-output RMSE, and detector accuracy, respectively.The experiments also assess transferability and memory and computation overhead; Pascal VOC is used for evaluation only.Source: §4 · p. 6

4.1. Model Accuracy

  • KTN performs on par with SPHCONV on Spherical MNIST, while both outperform other baselines on full-resolution Pano2Vid and Pascal VOC.Several other spherical CNN baselines face memory constraints that restrict their input resolution and reduce their accuracy on these realistic datasets.Source: §4.1 · p. 7
  • On Pano2Vid, SPHCONV has better RMSE, but KTN achieves better object classification accuracy on Pascal VOC.The authors attribute this difference to KTN’s parameter sharing across polar angles, which provides richer training information than SPHCONV’s independently trained kernels.Source: §4.1 · p. 7
  • KTN’s detector successfully detects objects despite distortion, but can fail when a very close object falls outside the field of view of perspective images.Source: §4.1 · p. 7

4.2. Transferability

  • KTN-TRANSFER performs almost identically to KTN and transfers across source models with the same base architecture, including models trained for different tasks.The evaluated transfers include Faster R-CNN models trained on COCO and Pascal VOC; the authors also report similar results for ImageNet classification and Pascal object detection.Source: §4.2 · pp. 7–8

4.3. Size and Speed

  • KTN adds 25% (14MB) to the source CNN, is orders of magnitude smaller than SPHCONV, and runs 26 times faster than SPHCONV.KTN has higher computational cost than EQUIRECTANGULAR, while the figure reports size and speed versus accuracy for VGG.Source: §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

KTN transfers perspective-trained CNNs to 360° imagery by transforming kernels for spherical distortion. The same KTN can transfer across source CNNs with the same architecture, and the paper reports improved performance, scalability, and transferability.

  • KTN transforms kernels from perspective-trained CNNs to account for distortion in 360° equirectangular images.Source: §5 · p. 8
  • A single KTN model can transfer to multiple source CNNs that share the same architecture.Source: §5 · p. 8
  • The authors report that KTN outperforms existing methods while providing scalability and transferability.Source: §5 · p. 8

A. KTN Architecture

KTN transforms source kernels into spherical convolution kernels and applies them at polar-angle-dependent rows. Its architecture shares kernels across nearby rows and uses a ResNet-like design; target kernel shapes are selected to cover back-projected receptive fields.

  • For each source CNN layer, KTN learns a function that transforms its kernel into a spherical convolution kernel.Source: A. KTN Architecture · p. 9
  • KTN shares a generated kernel across every five rows because kernels for adjacent rows are usually similar, reducing computation and model size.Source: A. KTN Architecture · p. 9
  • The KTN uses a ResNet-like architecture with row-dependent channel-wise projections and residual branches containing two depth-separable convolution blocks.Each block applies ReLU, pointwise convolution, ReLU, then depthwise convolution.Source: A. KTN Architecture · p. 9
  • Target kernel shape covers the back-projected receptive field at each polar angle, with odd dimensions and a maximum height and width below 65 pixels.Dilation increases the effective receptive field when needed.Source: A. KTN Architecture · p. 9
  • For Spherical Faster R-CNN, features are projected to the tangent plane before the RPN and detector networks are applied there.Source: A. KTN Architecture · p. 9

B. Experimental Details

This section provides experimental details omitted from the main paper.

B.1. Datasets

The supplementary material describes the evaluation datasets and baseline implementations. The datasets include spherical digit classification, 360° video, and projected Pascal VOC detection; baseline methods vary in their convolution, pooling, and input-resolution choices.

B.1. Datasets

  • Spherical MNIST backprojects digits into 160×80 equirectangular images, trains at random polar angles, and tests at nine angles from 8° to 72°.Digits are not rotated; accuracy averaged across polar angles is the evaluation metric.Source: §B.1 · p. 9
  • Pano2Vid uses 1,056 frames from three categories for training and 168 frames from Mountain Climbing for testing, with RMSE over final convolution outputs as its metric.Source: §B.1 · p. 9
  • Pascal VOC evaluation backprojects annotated boxes to 640×320 equirectangular images at five polar angles, zero-pads outside the source field of view, and uses detector accuracy.The dataset is evaluation-only; ground-truth boxes are used for ROI-pooling.Source: §B.1 · pp. 9–10

B.2. Baselines

  • Baseline models use three convolution layers for Spherical MNIST and VGG16-matched depth and kernels for Pano2Vid and Pascal VOC.For the latter datasets, max-pooling is removed and conv5 uses dilation factor two.Source: §B.2 · p. 10
  • The baseline set includes ordinary CNNs on equirectangular or cubemap projections, alongside spherical convolution methods and SphConv.Cubemap outputs for Pano2Vid and Pascal VOC are reprojected to equirectangular form.Source: §B.2 · p. 10
  • PROJECTED derives kernels by bilinear interpolation under the assumption that kernel transformation can be modeled by tangent-plane-to-sphere projection.Source: §B.2 · p. 10
  • S2CNN and SPHERICAL CNN require 1:1 input aspect ratios, so their azimuthal resolution is reduced; other methods use the common 2:1 format.Source: §B.2 · p. 10

B.3. Training Details

Training used ADAM for 40 epochs, with learning-rate decay and L2 regularization; batch sizes varied by method and dataset, and Pano2Vid KTN training took about a week.

  • KTN training on Pano2Vid took about a week on six AWS p3.8xlarge instances with V100 GPUs.Source: §B.3 · p. 11
  • All methods used ADAM for 40 epochs, with a learning rate initialized at 1.0×10^-3 and reduced tenfold after 20 epochs, plus L2 regularization at 5×10^-4.Source: §B.3 · p. 11
  • Batch sizes differed by method and dataset: Pano2Vid used 1 for S2CNN, 2 for Spherical CNN, and 4 otherwise; Spherical MNIST used 64 except 16 for S2CNN.Weights were randomly initialized from a normal distribution with standard deviation 0.01.Source: §B.3 · p. 11

C. Transferability on Pascal VOC

The Pascal VOC evaluation reports that a KTN trained on one VGG source can transfer to Faster R-CNN detection, with performance nearly unchanged across source models.

D. Model Accuracy versus Depth

On Pano2Vid with Faster R-CNN, the accuracy gap between KTN and the interpolation-based baselines grows at greater network depth, while errors increase for all methods.

E. Multiple Projections Baseline

KTN’s worst accuracy exceeds the best EQUIRECTANGULAR and CUBEMAP accuracy in the comparison; additional examples show both successful detections and failure cases.

E. Multiple Projections Baseline

F. Object Detection Examples

  • Examples on Pano2Vid and Pascal VOC show KTN detecting distorted objects by translating the source CNN to spherical data.Source: F. Object Detection Examples · p. 12
  • Pano2Vid failures include splitting one person into two detections when close objects exceed the perspective field of view, and missing a rare top view.The passage notes that 360° and perspective-image data distributions differ.Source: F. Object Detection Examples · p. 12
Loading 1812.03115v2…