PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Oriented Boundaries: From Image Segmentation to High-Level Tasks

Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Pablo Arbeláez, Luc Van Gool

arXiv:1701.04658v2cs.CV

TL;DR

Existing grouping benchmarks are limited by BSDS saturation and small scale, motivating methods that exploit larger datasets and high-capacity CNNs. COB learns multiscale oriented contours in one CNN pass and efficiently constructs region hierarchies with sparse boundaries. It achieves state-of-the-art low-level results and improves multiple downstream vision pipelines, while its box-proposal performance is less competitive because it was not designed specifically for bounding-box detection.

  • Problem

    BSDS is nearing saturation and has too few images to adequately train or evaluate current large-capacity grouping models.

  • Method

    COB uses a generic CNN to learn multiscale contour strength and orientation in one forward pass, then constructs hierarchical regions with a sparse boundary representation.

  • Results

    COB achieves state-of-the-art contours, orientations, and RGB/RGB-D segmentation, while improving object proposals, detection, semantic contours, and segmentation pipelines.

  • Takeaways & Limitations

    COB transfers improvements from deeper CNN backbones to perceptual grouping and avoids the globalization bottleneck in its supported tasks.

  • Takeaways & Limitations

    COB is less competitive for bounding-box proposals because it was not specifically designed for detecting bounding boxes.

Abstract

from arXiv · show

We present Convolutional Oriented Boundaries (COB), which produces multiscale oriented contours and region hierarchies starting from generic image classification Convolutional Neural Networks (CNNs). COB is computationally efficient, because it requires a single CNN forward pass for multi-scale contour detection and it uses a novel sparse boundary representation for hierarchical segmentation; it gives a significant leap in performance over the state-of-the-art, and it generalizes very well to unseen categories and datasets. Particularly, we show that learning to estimate not only contour strength but also orientation provides more accurate results. We perform extensive experiments for low-level applications on BSDS, PASCAL Context, PASCAL Segmentation, and NYUD to evaluate boundary detection performance, showing that COB provides state-of-the-art contours and region hierarchies in all datasets. We also evaluate COB on high-level tasks when coupled with multiple pipelines for object proposals, semantic contours, semantic segmentation, and object detection on MS-COCO, SBD, and PASCAL; showing that COB also improves the results for all tasks.

1 INTRODUCTION

COB addresses the limits of saturated BSDS-based grouping by combining multiscale oriented contour detection with efficient hierarchical segmentation. It achieves state-of-the-art contours, regions, and downstream-task improvements across challenging datasets.

  • Approach: COB converts generic CNNs into multiscale oriented contour detectors and builds hierarchical regions using a sparse boundary representation.The approach uses a single CNN forward pass for contour detection and combines contours into hierarchical segmentation structures.
  • Results: 0.8 seconds per image: COB produces state-of-the-art contours and regions on PASCAL and BSDS while remaining computationally efficient.The paper attributes efficiency to its sparse boundary representation for hierarchical regions.
  • Findings: Deeper CNN backbones improve COB, and oriented contours with CNN features make spectral globalization unnecessary in PASCAL.Removing globalization eliminates a computational bottleneck while preserving high-quality contours.
  • Results: COB achieves state-of-the-art performance across contour detection, orientation estimation, and generic RGB and RGB-D segmentation.Experiments include PASCAL Context, BSDS500, NYUD, and PASCAL VOC object contours.
  • Downstream applications: COB improves object proposals, object detection, semantic contours, and semantic segmentation when combined with recent high-level pipelines.The method also generalizes to MS-COCO without retraining and achieves super-human retinal vessel and optic-disc segmentation performance.

2 RELATED WORK

Prior contour and grouping systems combine multiscale cues, orientation, and deep learning in separate or costly components. COB integrates these cues in one CNN pass and uses high-level features to simplify hierarchical segmentation.

  • Contour detection: Early contour detectors relied on local gradients, zero crossings, and nonmaximum suppression, but richer color and texture features later improved boundary modeling.Machine-learning methods introduced learned classifiers and features for contour detection.
  • Deep learning: Deep contour methods use patch features, pretrained CNN representations, multiclass classification, or contour regression to exploit object-level information.These approaches established deep learning as a route to state-of-the-art contour detection.
  • COB distinction: COB combines multiscale information, contour orientation, and end-to-end deep learning in a single CNN forward pass.Its output is richer than a linear combination of cues from different scales.
  • Architecture: COB starts from generic pretrained CNN backbones and leverages their intermediate stages to generate multiscale contours.The architecture can use networks such as VGG or ResNet as its base CNN.
  • Hierarchical grouping: Unlike spectral grouping methods, COB uses sparse boundaries and CNN-derived contour strength and orientation to bypass computationally expensive globalization.This exploits the contour-region duality while incorporating global information from high-level CNN features.

3 DEEP MULTISCALE ORIENTED CONTOURS

COB uses intermediate CNN representations to learn multiscale contour strength and orientation in one forward pass, then combines these outputs into hierarchical segmentation. Its learned orientations are reported as more accurate and beneficial for region segmentation.

  • Multiscale contours: CNN depth supplies increasingly global information, while intermediate layers provide scale-specific features for multiscale contour and orientation detection.Deeper layers have lower resolution and larger receptive fields; shallower layers preserve finer localization.
  • Hierarchical segmentation: COB combines coarse and fine contour strengths with orientations non-linearly to construct a single hierarchical segmentation.The method uses CNN outputs rather than only a linear combination of multiscale cues.
  • Multiscale contours: COB supervises side activations at multiple stages to produce intermediate contour maps at different resolutions.The base CNN is divided into stages treated as scales, and each stage’s final layer receives supervision.
  • Multiscale contours: Fine-scale outputs provide better localization, whereas coarse-scale outputs yield less noisy contour detections.Trainable linear combinations form fine-scale and coarse-scale outputs from the side activations.
  • Contour orientation learning: COB extends the contour-strength CNN with K orientation-specific branches that classify contour pixels into orientation bins across M scales.Each branch uses intermediate convolutional feature maps, while the pre-trained base-network weights remain frozen during sub-network training.
  • Contour orientation learning: With K = 8 and M = 5, orientation responses are regressed in parallel and transformed into an orientation map using bin angles and weighted averaging of neighboring responses.Pixels without an orientation response receive random values between 0 and π to avoid biasing orientations.
  • Contour orientation learning: Learned orientations are significantly more accurate than orientations computed with local gradient filters and lead to better region segmentations.The comparison is made against orientation estimation used in prior methods.

4 FAST HIERARCHICAL REGIONS

The method builds hierarchical regions from multiscale contours using a sparse boundary representation that makes segmentation operations more efficient. It combines single-pass CNN outputs across detail levels and learned orientations to construct faster, more accurate hierarchies.

  • Hierarchical regions: Ultrametric Contour Maps represent hierarchies by progressively erasing boundaries in increasing strength order, producing partitions at different granularities.Boundary strength indicates confidence, and removing weaker boundaries merges neighboring regions.
  • Sparse boundary representation: Sparse boundaries store neighboring-region pairs, boundary strengths, and occupied coordinates instead of scanning a largely empty boundary grid.This reduces memory use and enables operations on specific boundary pieces through lookup and coordinate scans.
  • Multiscale contours: Different CNN output levels provide high- and low-level contour details in a single forward pass, avoiding separate processing at multiple image resolutions.The resulting outputs are used to obtain hierarchies focused on different feature scales.
  • Efficient construction: The sparse representation supports operations across all thousands of contour strengths rather than only a small subset, at a fraction of the original computational cost.The original multiscale framework required manipulating only a few dozen strengths because hierarchy operations and scale projection took seconds.
  • Oriented contours: Learned contour orientations are used in the Oriented Watershed Transform to further improve hierarchical-region performance.Orientation information supplements contour strength during hierarchy computation.

5 EXPERIMENTS ON LOW-LEVEL APPLICATIONS

COB is evaluated across low-level contour detection, segmentation, orientation estimation, downstream RGB-D settings, and efficiency analyses. The experiments show strong accuracy, benefits from trained orientations and suitable CNN depth, and substantial computational savings.

  • Control experiments and ablations: ResNet50 with trained orientations achieves the best boundary and region results, while deeper ResNets provide negligible accuracy gains at substantially lower speed.The ablations also show that globalization can be removed when trained orientations are used, improving efficiency.
  • Generic image segmentation: COB outperforms prior methods by a considerable margin on PASCAL Context for both boundaries and regions, while achieving state-of-the-art results on BSDS500.The smaller BSDS margins are attributed to the benchmark approaching human contour-detection performance.
  • Generic image segmentation: COB distinguishes internal contours from external semantically meaningful boundaries in qualitative hierarchical-contour results.This qualitative analysis concerns the structure of the produced hierarchical contours.
  • RGB-D applications: On NYUD, depth embeddings improve contour accuracy when RGB alone is insufficient, and adding orientations plus the UCM pipeline raises performance from 0.745 to 0.784.The comparison is between ResNet50-RGB-HHA and COB-RGB-HHA configurations.
  • Efficiency analysis: COB reduces the comparison with previous state-of-the-art timing from 24.37 to 0.79 seconds by combining one-pass multiscale detection, sparse boundaries, and orientation estimation.Sparse boundaries reduce UCM and OWT time from 11.58 to 1.63 seconds, while the network forward pass computes contour strength and eight orientations in 0.28 seconds.

6 EXPERIMENTS ON HIGH-LEVEL APPLICATIONS

COB improves high-level vision pipelines by providing region hierarchies and oriented boundaries that transfer across object proposals, semantic boundaries, segmentation, and detection tasks.

  • 6 EXPERIMENTS ON HIGH-LEVEL APPLICATIONS: Across the evaluated high-level pipelines, COB cooperates with existing approaches and improves their performance.The experiments cover object proposals, semantic contours and segmentation, and object detection.
  • 6.1 Object Proposals: COB region hierarchies produce segmented object proposals with at least a +13% absolute improvement over the second-best technique on PASCAL VOC’12 Segmentation.On MS-COCO, without retraining on COCO images, COB remains competitive, with only very recent techniques performing better at low proposal counts.
  • 6.1 Object Proposals: COB is less competitive for bounding-box proposals because it was designed to detect segmented object proposals rather than bounding boxes.The comparison also notes that RPN trained on VOC’07 does not generalize well to COCO classes.
  • 6.2 Semantic Boundaries and Semantic Segmentation: Combining COB boundaries with dilated-convolution semantic segmentation achieves state-of-the-art semantic-boundary results on SBD in both benchmark metrics.The combination, COB-dil, improves over evaluating the semantic-segmentation results directly as contours.
  • 6.2 Semantic Boundaries and Semantic Segmentation: Using COB UCMs as high-recall superpixels refines semantic segmentation mainly at boundary locations and reduces noisy detections where COB superpixels are present.The improvement is more moderate for PSPNet, mainly because of reduced false detections.
  • 6.3 COB Object Proposals for Object Detection: COB segmented proposals can be converted into bounding boxes and supplied to Fast-RCNN while keeping the original pipeline’s hyperparameters unchanged.The experiments use VOC’07 detection data and replace Selective Search proposals with COB proposals.

7 CONCLUSIONS

COB detects multiscale oriented contours in one CNN forward pass and efficiently builds region hierarchies using a sparse boundary representation. It achieves state-of-the-art results across low- and high-level vision tasks while transferring improvements from CNN backbones.

  • 7 CONCLUSIONS: COB detects contours at multiple scales and orientations in a single convolutional-neural-network forward pass.A sparse boundary representation enables efficient construction of region hierarchies from these detections.
  • 7 CONCLUSIONS: COB achieves state-of-the-art performance by a significant margin for contour detection, contour orientation estimation, and generic RGB and RGB-D image segmentation.The approach also avoids the globalization step that was a speed bottleneck in previous methods.
  • 7 CONCLUSIONS: Using two different CNN base architectures demonstrates COB’s modularity and suggests that future backbone improvements can transfer to perceptual grouping.The conclusion further reports improved results when COB is combined with pipelines for object proposals, object detection, semantic contours, and semantic segmentation.
  • 7 CONCLUSIONS: The authors publicly release code, CNN models, precomputed results, dataset splits, and benchmarks.The release is provided through the project website.
Loading 1701.04658v2…