PaperlayerView exact revision on arXiv

Source-linked AI summary

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, Jason Yosinski

arXiv:1807.03247v2cs.CVcs.LGstat.ML

TL;DR

The paper asks why ordinary convolution struggles with seemingly simple coordinate transforms and investigates this question through toy and applied tasks. It introduces CoordConv, which adds explicit coordinate channels, and reports improved results across generative modeling, detection, and Atari reinforcement learning.

  • Problem

    CNNs are not quite the right model for transforming between dense Cartesian and sparse pixel-based spatial representations, despite their success on many image tasks.

  • Method

    The paper defines the Not-so-Clevr toy dataset, analyzes coordinate-transform tasks, and introduces CoordConv by adding hard-coded coordinate channels to convolutional inputs.

  • Results

    CoordConv improves performance across diverse tasks, including less GAN mode collapse, better Faster R-CNN boxes and scores, and significantly higher Atari scores on some games without significantly worse scores.

  • Takeaways & Limitations

    CoordConv provides a simple drop-in convolutional replacement that can preserve translation invariance or learn translation dependence as required by the task.

Abstract

from arXiv · show

Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional neural networks may be appropriate. In this paper we show a striking counterexample to this intuition via the seemingly trivial coordinate transform problem, which simply requires learning a mapping between coordinates in (x,y) Cartesian space and one-hot pixel space. Although convolutional networks would seem appropriate for this task, we show that they fail spectacularly. We demonstrate and carefully analyze the failure first on a toy problem, at which point a simple fix becomes obvious. We call this solution CoordConv, which works by giving convolution access to its own input coordinates through the use of extra coordinate channels. Without sacrificing the computational and parametric efficiency of ordinary convolution, CoordConv allows networks to learn either complete translation invariance or varying degrees of translation dependence, as required by the end task. CoordConv solves the coordinate transform problem with perfect generalization and 150 times faster with 10--100 times fewer parameters than convolution. This stark contrast raises the question: to what extent has this inability of convolution persisted insidiously inside other tasks, subtly hampering performance from within? A complete answer to this question will require further investigation, but we show preliminary evidence that swapping convolution for CoordConv can improve models on a diverse set of tasks. Using CoordConv in a GAN produced less mode collapse as the transform between high-level spatial latents and pixels becomes easier to learn. A Faster R-CNN detection model trained on MNIST showed 24% better IOU when using CoordConv, and in the RL domain agents playing Atari games benefit significantly from the use of CoordConv layers.

1 Introduction

The paper identifies a generic difficulty for ordinary convolutional stacks in transforming between Cartesian coordinates and pixel-based spatial representations. It introduces CoordConv as a coordinate-aware extension and evaluates it from toy problems through broader applications.

  • Motivation: Ordinary convolutional networks struggle to learn coordinate transforms between dense Cartesian and sparse pixel-based representations.The authors contrast this difficulty with CNN success on tasks such as image classification.
  • Approach: The paper defines the Not-so-Clevr toy dataset of randomly positioned squares and uses progressively harder experiments to analyze the problem.The experiments begin with simple supervised settings before considering more complex tasks and applications.
  • Approach: CoordConv gives convolutional filters access to Cartesian position by adding hard-coded coordinate channels to the input.The operation is presented as a simple modification requiring only a few extra implementation lines.
  • Experiments: Supervised rendering remains difficult for convolution but is trivial for CoordConv when networks paint full images from square-center coordinates.This task extends coordinate classification while remaining fully supervised.
  • Results: CoordConv improves diverse downstream tasks, including reduced mode collapse in GANs and better object detection, while Atari agents score significantly higher on some games and never significantly worse.The reported evidence spans generative modeling, Faster R-CNN detection, and reinforcement learning.

2 Not-so-Clevr dataset

Not-so-Clevr provides a compact coordinate-transform benchmark, while CoordConv augments convolution with explicit spatial coordinates. The extension preserves parameter sharing and computational efficiency while allowing learned translation dependence.

  • Dataset: Not-so-Clevr contains 9 × 9 grayscale squares placed on a 64 × 64 canvas, with 3,136 possible center positions.Square centers occupy the valid 56 × 56 region so the full square remains within the canvas.
  • Dataset: Each example stores a Cartesian center coordinate Ci, a one-hot center-pixel representation Pi, and the rendered canvas image Ii.These fields support transformations between coordinate, sparse pixel, and image representations.
  • Dataset: The dataset uses uniform 80/20 and quadrant train/test splits, with the latter holding out one quadrant for testing.The quadrant split tests generalization beyond the spatial regions represented in training.
  • CoordConv layer: CoordConv concatenates constant coordinate channels to the incoming representation before applying a standard convolution.The basic channels encode i and j coordinates scaled to [−1, 1], with an optional radius channel r.
  • Properties: A CoordConv layer uses (c + d)c′k^2 weights instead of cc′k^2, adding only the coordinate-channel connections.The coordinate weights remain shared across spatial positions.
  • Properties: CoordConv can retain translation invariance or learn translation dependence depending on its coordinate weights.Zero coordinate weights make it mathematically equivalent to ordinary convolution; nonzero weights introduce task-dependent spatial dependence.

4 Supervised Coordinate tasks

The supervised coordinate tasks expose a sharp mismatch between ordinary convolution and coordinate transforms, while CoordConv solves both directions and extends to rendering.

  • 4.1 Supervised Coordinate Classification: Given an (x, y) coordinate, supervised coordinate classification requires painting the corresponding output pixel.The task treats each pixel as a class and motivates testing whether networks can map Cartesian coordinates to pixel representations.
  • 4.1 Supervised Coordinate Classification: 86% was the best convolutional test accuracy on the uniform split, while convolutional models failed to generalize on the quadrant split.The uniform split places test pixels near multiple training pixels, yet perfect test accuracy was never achieved; larger models could memorize training data.
  • 4.1 Supervised Coordinate Classification: 7.5k parameters and 10–20 seconds enabled CoordConv to attain perfect performance on both coordinate-classification splits.The comparable convolutional model required about 200k parameters and over an hour to reach 86% accuracy.
  • 4.2 Supervised Coordinate Regression: Coordinate regression reverses the problem by predicting real-valued (x, y) coordinates from a one-hot pixel input.Convolutional architectures showed split-dependent generalization, whereas a 900-parameter CoordConv model provided a simpler alternative.
  • 4.3 Supervised Rendering: The supervised rendering task asks networks to produce a 64 × 64 image with a square centered at the supplied (x, y) location, again favoring CoordConv over convolution.This extends the single-pixel coordinate transform to painting a full image.

5 Applicability to Image Classification, Object Detection, Generative Modeling, and Reinforcement Learning

The paper tests whether convolution’s coordinate-transform difficulty affects other applications. CoordConv helps on object detection, generative modeling, and some Atari games, while providing no significant ImageNet classification benefit.

  • Image Classification: CoordConv does not significantly improve ImageNet classification, increasing ResNet-50 Top-5 accuracy by only 0.04% on average.The difference was not statistically significant, and performance did not worsen.
  • Object Detection: 24% higher test IOU was obtained by Faster R-CNN on scattered MNIST digits when using CoordConv.Object detection naturally transforms pixel-space inputs into Cartesian-space bounding boxes.
  • Generative Modeling: CoordConv GAN samples covered the 2D Cartesian space better than convolutional GAN samples while using 7% of the convolutional GAN’s parameters.The convolutional GAN collapsed a two-dimensional distribution onto a one-dimensional manifold.
  • Generative Modeling: CoordConv produced smooth geometric translation and deformation during LSUN latent-space interpolation, unlike convolutional models’ fading frozen objects.The models produced qualitatively comparable samples when drawing randomly.
  • Reinforcement Learning: In A2C-trained Atari agents, CoordConv improved performance in 6 of 9 games, matched convolution in 2, and was slightly worse in 1.Adding CoordConv to an Ape-X implementation produced no immediate difference.

6 Conclusions and Future Work

The paper concludes that CoordConv addresses CNNs’ coordinate-transform difficulty through added coordinate channels and can improve diverse applications. It identifies broader evaluation of these benefits as future work.

  • Future Work: Future work will evaluate CoordConv on larger datasets and in translation perturbations, relational reasoning, language, video prediction, and spatial transformer settings.The authors frame these directions as further evaluation rather than settled conclusions.
  • Conclusions: The supervised experiments show that convolution struggles with coordinate-transform tasks while CoordConv models can perform well with substantially fewer parameters.A working CoordConv model used 7,553 parameters versus 50k–1.6M for the best convolutional models.

S2 Further Supervised Coordinate Classification details

The supplementary coordinate-classification experiments use extensive convolutional hyperparameter searches and show that CoordConv converges quickly with smooth, precise predictions. Convolutional models remain less accurate and less consistent near neighboring train and test coordinates.

  • Experimental Setup: Deconvolution models were swept over filter sizes, channel multipliers, learning rates, and weight decay using Adam and cross-entropy training.The searched filter sizes were 2, 3, and 4, with channel multipliers 1, 2, and 3.
  • Experimental Setup: CoordConv models converged perfectly with only three learning rates and required no learning-rate schedules.The coordinate-classification CoordConv models converged in 10 seconds.
  • Results: CoordConv produced accurate and smooth probability mass across neighboring training and test coordinates, whereas deconvolution showed artifacts.The comparison examines full 64 × 64 logit maps and zoomed local regions.
  • Architectures: The regression architectures differed by split: the best uniform-split convolution model used convolution, max pooling, and fully connected layers, while the quadrant model used strided convolutions and global pooling.The supplied architecture description is specific to convolutional regression baselines.

S4 Further Supervised Rendering details

The supervised rendering task compares convolution and CoordConv on painting a square at a specified coordinate. CoordConv retains the strong generalization and training-speed advantage observed in the other supervised tasks.

  • Training: The rendering loss uses pixelwise sigmoid cross-entropy, while mean squared error performed even more weakly.Deconvolution models received heavy hyperparameter sweeps and learning-rate annealing; CoordConv settings were easier to find.
  • Results: CoordConv learned smooth and precise pixelwise probability distributions on uniform- and quadrant-split test samples.The displayed comparisons include logits and sigmoid-transformed pixelwise probabilities.
  • Uniform Split: On the uniform split, convolution outputs were roughly correct, whereas CoordConv outputs were precisely correct with smooth logit maps.The comparison covers three test samples.
  • Quadrant Split: On the quadrant split, convolution outputs mostly failed while CoordConv outputs remained precisely correct with smooth logit maps.The quadrant split tests a more difficult generalization setting.

S5 Further ImageNet classification details

The ImageNet experiment adds a small CoordConv input layer to ResNet-50, producing only a tiny average change in classification performance.

  • The CoordConv variant adds one initial layer that receives RGB, i, j, and distance-to-center r channels, with negligible parameter increase.
  • CoordConv performs better on two of three measures, while the Top-5 accuracy improvement has p = .11 in a one-sided t-test.
  • The small classification improvement matches the expectation that image classification requires straightforward translation invariance and depends less on object location.

S6 Further object detection details

The object-detection experiment evaluates Faster R-CNN on randomly placed MNIST digits, comparing regular convolution with CoordConv across test-set box metrics.

  • The dataset contains five randomly rescaled and placed MNIST digits on a 64 × 64 canvas, with some digits partially outside the canvas.It uses 9,000 training images and 1,000 test images.
  • CoordConv improves every reported detection metric, including train IOU and two test IOU measures.The metrics compare sampled positive boxes, ten selected test boxes, and the best-scored box with its closest ground truth.
  • 24%: the average test IOU improves with CoordConv relative to regular convolution.
  • Faster R-CNN uses box sampling during training and box non-maximum suppression during testing, with no downstream classification task.

S7 Further generative modeling details

The generative-model experiments compare regular convolution with CoordConv in GANs and VAEs on colored shapes and LSUN bedrooms, emphasizing coordinate-aware latent-to-image transformations.

  • Colored-shape generative models: The colored-shape dataset contains 50,000 64 × 64 images with one red and one blue non-overlapping object at random positions.Each object is either a circle or square.
  • Colored-shape generative models: The CoordConv GAN adds coordinate channels at the generator’s first layer, reducing generator parameters from 6,413,315 to 444,931.The discriminator is shared between the regular and CoordConv GANs in the primary comparison.
  • Colored-shape generative models: Latent interpolation is performed by linearly interpolating between two uniformly sampled noise vectors with an α factor.
  • Colored-shape generative models: Entropy analysis finds CoordConv closer to the data in objects’ spatial coverage but with more mode collapse in relative positions.
  • Colored-shape generative models: CoordConv GAN interpolations show smooth circular motion and relative object-position changes, including extrapolated motion beyond the training frame.
  • VAEs on colored shapes: CoordConv VAEs provide similar benefits to GANs: smooth latent-to-pixel geometric changes and fuller spatial coverage, including extrapolation beyond frame borders.
  • GANs on LSUN: On LSUN bedrooms, regular GAN interpolations show objects appearing and disappearing, whereas CoordConv interpolations show translation and other geometric transformations.

S8 Further reinforcement learning details

The reinforcement-learning experiments use OpenAI Baselines defaults and report average end-of-game scores across ten runs for each Atari game.

  • The Atari experiments use the OpenAI Baselines implementation with default parameters.
  • Scores are averaged at the end of each game over 10 runs for every model.
  • Table S7 reports final scores and p-values for all evaluated games.

S9 The CoordConv layer implementation

The CoordConv implementation adds coordinate channels to an input tensor before applying an ordinary convolution. Coordinates are generated, normalized, optionally augmented with a radial channel, concatenated with the input, and passed through Conv2D.

  • S9 The CoordConv layer implementation: AddCoords stores the spatial dimensions and whether to include a radial coordinate channel.The layer records x_dim, y_dim, and with_r for coordinate construction.
  • S9 The CoordConv layer implementation: The implementation constructs tiled x- and y-coordinate grids for each batch element.It creates ranges, expands dimensions, tiles them across the batch, and forms coordinate channels with matrix multiplication and expansion.
  • S9 The CoordConv layer implementation: The coordinate channels are cast to float32, scaled to [0,1], and then remapped to [-1,1].Both x and y channels use their corresponding spatial dimension for normalization.
  • S9 The CoordConv layer implementation: When enabled, the radial channel is derived from the squared coordinate channels and concatenated with the existing result.The implementation appends rr after adding the coordinate channels.
  • S9 The CoordConv layer implementation: CoordConv applies AddCoords to the input and then feeds the augmented tensor into a standard Conv2D layer.The call method performs coordinate augmentation, convolution, and returns the output.
Loading 1807.03247v2…