Source-linked AI summary
SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360 degree Images
Yeonkun Lee, Jaeseok Jeong, Jongseob Yun, Wonjune Cho, Kuk-Jin Yoon
TL;DR
Mapping omnidirectional images to Euclidean representations can distort shapes and lose continuity, making CNN processing difficult. SpherePHD uses an icosahedral spherical polyhedron with tailored convolution and pooling methods. Across the reported classification, detection, and segmentation tasks, the paper reports more uniform MNIST accuracy, better performance on tilted-image detection, and higher segmentation accuracy than other representations.
Problem
Euclidean representations of omnidirectional images can introduce shape distortion from nonuniform spatial resolving power and lose continuity.
Source: §1 · pp. 1–2Method
Results
SpherePHD has relatively uniform MNIST accuracy, works better on tilted SYNTHIA detection images, and outperforms other methods on both semantic-segmentation datasets.
Source: §4.1 · p. 6, §4.2 · p. 7, §4.3 · p. 8Takeaways & Limitations
The proposed representation and CNN methods were demonstrated on classification, detection, and semantic-segmentation tasks using MNIST, SYNTHIA, and Stanford2D3D.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Omni-directional cameras have many advantages overconventional cameras in that they have a much wider field-of-view (FOV). Accordingly, several approaches have beenproposed recently to apply convolutional neural networks(CNNs) to omni-directional images for various visual tasks.However, most of them use image representations defined inthe Euclidean space after transforming the omni-directionalviews originally formed in the non-Euclidean space. Thistransformation leads to shape distortion due to nonuniformspatial resolving power and the loss of continuity. Theseeffects make existing convolution kernels experience diffi-culties in extracting meaningful information.This paper presents a novel method to resolve such prob-lems of applying CNNs to omni-directional images. Theproposed method utilizes a spherical polyhedron to rep-resent omni-directional views. This method minimizes thevariance of the spatial resolving power on the sphere sur-face, and includes new convolution and pooling methodsfor the proposed representation. The proposed method canalso be adopted by any existing CNN-based methods. Thefeasibility of the proposed method is demonstrated throughclassification, detection, and semantic segmentation taskswith synthetic and real datasets.
1. Introduction
Omnidirectional images provide a wide view, but mapping them to ERP grids can distort shapes and disrupt continuity, complicating visual processing. SpherePHD represents these images on a spherical polyhedron and introduces convolution and pooling methods for CNNs.
1. Introduction
- ERP’s nonuniform spatial resolving power distorts images, especially near the poles, and camera tilt can produce a fluctuating horizon.The varying convolution domain also makes identical kernels inadequate across the ERP image.Source: §1 · pp. 1–2
- ERP’s loss of continuity at image borders can make a single object split across the border appear as two detections.The paper illustrates this with a car detected as two objects with different IDs.Source: §1 · p. 2, Figure 2 · p. 2
- Cube maps reduce spatial-resolving-power variance and are less affected by rotation than ERP, but retain face-center-to-edge variance and kernel-orientation ambiguity on top and bottom faces.The paper identifies these as remaining limitations of cube-map CNN representations.Source: §1 · p. 2
- SpherePHD projects images onto an icosahedral spherical polyhedron, with lower spatial-resolving-power variance and distortion than other representations.Its rotational symmetry supports rotation-invariant processing, and its continuity property removes image borders where discontinuities arise.Source: §1 · p. 2
- The paper introduces convolution and pooling methods for CNNs on SpherePHD and compares representations on classification, detection, and semantic-segmentation tasks.The comparisons use spherical MNIST, SYNTHIA, and Stanford2D3D datasets created by projecting the original datasets onto the spherical polyhedron.Source: §1 · p. 2
2. Related works
Prior work adapts ERP images, uses cube-based representations, or draws on geographic map projections to address issues in omnidirectional CNNs. The paper positions its representation as further reducing spatial-resolving-power variance without leaving the spatial domain.
2. Related works
- The related-work section frames prior approaches as studies of CNNs on omnidirectional images.It introduces ERP-based methods and other representations as the main categories discussed.Source: §2 · p. 2
2.1. ERP-based methods
- ERP-based approaches adjust sampling, partition images into NFoV views, or use distortion-aware kernels; NFoV sampling improves accuracy but does not use the whole omnidirectional view.Camera-tilt post-processing requires knowledge of camera orientation, while this paper replaces ERP-based representation to address ERP distortion.Source: §2.1 · pp. 2–3
2.2. Other representations for 360◦images
- Cube maps address several ERP problems, and face padding lets convolution cross adjacent faces, but spatial-resolving-power variance remains between face centers and edges.Equi-angular cube projection addresses that variance; another approach transforms from Euclidean S2 space to the SO(3) 3D rotation group.Source: §2.2 · p. 3
2.3. Representations in geography
- The paper’s proposed spherical polyhedron representation is likened to the Dymaxion map projection method.The related-work section also likens Yang et al. [22] to Mercator and Coors et al. to Hammer.Source: §2.3 · p. 3
3. Proposed method
The method projects omnidirectional images onto an icosahedral spherical polyhedron and designs convolution, pooling, and weight-assignment rules for CNNs on that representation. SpherePHD is chosen to reduce pixel-area irregularity while preserving spherical continuity and rotational properties.
3. Proposed method
- The method projects omnidirectional images onto a spherical polyhedron, which the paper describes as having less input-image irregularity than ERP or other representations.The proposed representation is intended for application to a CNN structure.Source: §3 · p. 3
3.1. Definition of irregularity
- The irregularity measure uses each pixel’s effective area on the unit sphere, its log-scaled ratio to the geometric-mean area, and the root-mean-square of those ratios.The log-scaled deviations sum to zero under the geometric-mean definition.Source: §3.1 · p. 3
3.2. Spherical polyhedron
- An icosahedron is selected because its greater number of faces can yield lower irregularity than the cube map’s six-face geometry; Figure 4 shows pixel-area variance decreasing with more equi-faced regions.Cube-map irregularity converges early as its faces are subdivided, indicating that intrinsic geometry matters more than subdivision count.Source: §3.2 · p. 4, Figure 4 · p. 4
3.3. SpherePHD: Icosahedral spherical polyhedron representation
- SpherePHD projects omnidirectional images onto a geodesic icosahedral polyhedron, yielding lower effective-pixel-area variance than the cube map and avoiding ERP’s sinusoidal fluctuation effects.The representation uses triangular pixels on the tessellated polyhedron.Source: §3.3 · p. 4
3.4. SpherePHD convolution and pooling
- Convolution kernels are centered on the pixel of interest, adapt to adjacent triangles’ orientations, and apply across pixels; pooling reduces subdivision by reversing the triangle-splitting process.The described example subdivides each triangle into four smaller triangles for pooling.Source: §3.4 · pp. 4–5
3.5. Kernel weight assignment
- Two pole-defined orientations use kernel shapes with shared weights ordered as 180° rotations, with the stated aim of learning rotation-invariant properties.The authors compare this weight assignment with non-rotated assignments in Section 4.1.Source: §3.5 · p. 5
3.6. CNN Implementation
- The implementation maps neighboring SpherePHD pixel values into stacked tensors so conventional 2-D convolution with a 1 × n kernel mimics spherical convolution.Convolution preserves the subdivision; pooling produces the next lower subdivision, and the method uses conventional CNN implementations.Source: §3.6 · pp. 5–6
4. Experiments
Experiments evaluated SpherePHD, cube map, and ERP representations on MNIST classification, SYNTHIA vehicle detection, and semantic segmentation on SYNTHIA and Stanford2D3D. Outcomes varied with task and image orientation, while SpherePHD performed better on tilted detection and both segmentation datasets.
4.1. MNIST classification
- The experiments used position- and orientation-augmented MNIST projections and a shared fully convolutional network structure, with representation-specific convolution and pooling methods.Training and test sets were augmented to 1,200k and 700k images, respectively, and image resolutions and parameter scales were controlled for comparison.Source: §4.1 · p. 6
- MNIST accuracy varied with position: SpherePHD was relatively uniform across latitude, while high-irregularity regions matched accuracy decreases and ERP also dropped near image boundaries.The latitude and longitude samples were uniformly distributed; the ERP longitude drop was additionally attributed to boundary discontinuity.Source: §4.1 · pp. 6–7, Figure 9 · p. 7, Figure 10 · p. 7
4.2. SYNTHIA vehicle detection
- SYNTHIA detection compared no-rotation and randomly rotated 360° scenes, using similar YOLO-based architectures and bounding circles to accommodate varied orientations.The dataset combines front, right, left, and back views captured from the same camera center.Source: §4.2 · p. 7
- On unrotated SYNTHIA detection ERP was more accurate, but on tilted detection SpherePHD performed better; SpherePHD also outperformed the alternatives on both semantic-segmentation datasets.The detection comparison used 18k training and 4.5k test images without tilt, and 180k training and 45k test images with tilt.Source: §4.2 · p. 7, §4.3 · p. 8, Table 2 · p. 8
4.3. Semantic Segmentation
- Semantic segmentation used SYNTHIA and real-world Stanford2D3D data, randomly tilted for augmentation, with a CNN autoencoder and max unpooling.Evaluation used average class accuracy and overall pixel accuracy; the former helps account for classes with few pixels.Source: §4.3 · p. 7
5. Conclusion
The paper addresses shape distortion from nonuniform resolving power and discontinuities in 360° image representations. It proposes SpherePHD and associated CNN operations, then demonstrates feasibility across classification, detection, and segmentation tasks.
5. Conclusion
- The paper identifies nonuniform resolving power and image-border discontinuities as sources of shape distortion in existing 360° representations.These problems motivate the proposed representation and CNN operations.Source: §5 · p. 8
- SpherePHD projects images onto an icosahedron-derived spherical polyhedron with less irregularity than ERP and cube map representations.The paper also introduces convolution and pooling methods for applying CNNs to SpherePHD images.Source: §5 · p. 8
- The authors demonstrate feasibility on MNIST, SYNTHIA, and Stanford2D3D through classification, detection, and semantic-segmentation tasks.The conclusion includes semantic-segmentation results in Table 3.Source: §5 · p. 8, Table 3 · p. 8