Source-linked AI summary
Salient Object Detection: A Discriminative Regional Feature Integration Approach
Huaizu Jiang, Zejian Yuan, Ming-Ming Cheng, Yihong Gong, Nanning Zheng, Jingdong Wang
TL;DR
Salient object detection needs an alternative to fragile heuristic integration of saliency cues. The paper learns a Random Forest regressor to map regional features from multi-level segmentations to saliency scores and fuses the resulting maps. DRFI consistently outperforms state-of-the-art methods across six benchmark datasets, while remaining broadly generalizable and computationally efficient as described by the paper.
Problem
Heuristic saliency-feature integration is fragile and generalizes poorly, motivating a supervised method for combining high-dimensional regional features.
Method
DRFI uses multi-level image segmentation and a Random Forest regressor to map regional contrast, property, and backgroundness features to saliency scores, then fuses maps across levels.
Results
DRFI consistently outperforms state-of-the-art algorithms on all six benchmark datasets with large margins in AUC, precision-recall, and ROC comparisons.
Takeaways & Limitations
The learned regressor automatically integrates high-dimensional features, selects discriminative ones, and generalizes to other datasets despite training on a small image set.
Takeaways & Limitations
Because DRFI mainly relies on regional contrast and backgroundness, it may fail on cluttered scenes and objects resembling or occupying much of the background.
Abstract
from arXiv · showhide
Salient object detection has been attracting a lot of interest, and recently various heuristic computational models have been designed. In this paper, we formulate saliency map computation as a regression problem. Our method, which is based on multi-level image segmentation, utilizes the supervised learning approach to map the regional feature vector to a saliency score. Saliency scores across multiple levels are finally fused to produce the saliency map. The contributions lie in two-fold. One is that we propose a discriminate regional feature integration approach for salient object detection. Compared with existing heuristic models, our proposed method is able to automatically integrate high-dimensional regional saliency features and choose discriminative ones. The other is that by investigating standard generic region properties as well as two widely studied concepts for salient object detection, i.e., regional contrast and backgroundness, our approach significantly outperforms state-of-the-art methods on six benchmark datasets. Meanwhile, we demonstrate that our method runs as fast as most existing algorithms.
1 INTRODUCTION
The paper frames salient object detection as a regression problem to replace fragile heuristic feature integration with supervised, discriminative regional feature integration. Its multi-level segmentation, learned region scoring, and score fusion aim to improve generalization and benchmark performance.
- Motivation: Salient object detection extends visual saliency from predicting eye fixations to identifying image regions containing salient objects.The task supports applications including object detection, image compression, cropping, collage, and dominant color detection.
- Motivation: Hand-crafted rules for combining saliency cues are fragile and do not consistently generalize across benchmark datasets.Existing approaches commonly combine conspicuity maps from color, texture, shape, and other cues using manually designed integration rules.
- Approach: The proposed pipeline segments images at multiple levels, regresses regional features to saliency scores, and fuses the resulting maps.A Random Forest regressor performs the regional saliency computation before multi-level fusion.
- Contribution: DRFI directly learns to integrate regional feature vectors and select discriminative features instead of relying on heuristic saliency integration.The approach uses generic region properties together with regional contrast and backgroundness.
- Contribution: The method consistently outperforms state-of-the-art algorithms on all six benchmark datasets with large margins.The learned regressor also shows generalization to other datasets despite training on a small image set.
2 RELATED WORK
Prior salient object detection methods largely use unsupervised contrast, uniqueness, and object or background priors, while supervised methods learn feature combinations. DRFI differs by learning a nonlinear, data-driven regional integration that directly produces the saliency map.
- Unsupervised approaches: Unsupervised methods commonly characterize saliency through center-surround contrast and combinations of low-level visual features.Related approaches use pixel or regional contrast, color histograms, information theory, and hierarchical segmentations.
- Unsupervised approaches: Other unsupervised approaches model saliency using soft region assignments, low-rank recovery, submodularity, and priors such as centrality, objectness, and backgroundness.These methods seek to capture uniqueness or incorporate assumptions about salient objects and image backgrounds.
- Supervised approaches: Supervised methods have learned linear fusion weights or used Random Forests for related saliency tasks, but prior approaches differ in scope and output.The cited Random Forest method localizes salient objects on thumbnail images, while another uses saliency only to guide object-recognition sampling.
- DRFI: DRFI extends regional contrast into a contrast vector and automatically combines high-dimensional saliency features without assuming a feature-space form.Unlike a cited coarse-estimation approach, DRFI directly outputs the saliency map.
- DRFI: The paper positions DRFI as a supervised salient object detector designed specifically for salient object detection rather than eye-fixation prediction.Its learned feature fusion is distinguished from related discriminative fusion work in image classification.
3 IMAGE SALIENCY COMPUTATION
The DRFI computation pipeline uses multi-level image segmentations, predicts saliency for each region from learned regional features, and fuses region scores across levels into a final map.
- Pipeline: The pipeline consists of multi-level segmentation, regional saliency computation, and multi-level saliency fusion.The framework is illustrated in Fig. 1.
- Multi-level segmentation: Multi-level segmentation represents an image with M decompositions generated using different graph-based segmentation parameters.Each segmentation provides a distinct region partition of the same image.
- Multi-level segmentation: Multiple segmentations improve robustness because no current low-level segmentation algorithm can reliably isolate the salient object.They also provide many training samples for the learning framework.
- Regional saliency computation: Each region is represented by regional contrast, regional property, and regional backgroundness features, then passed to a learned Random Forest regressor for scoring.The regressor maps the feature vector x to a regional saliency score.
- Multi-level saliency fusion: At each segmentation level, regional scores are assigned to the regions' contained pixels to form level-specific saliency maps.These maps are subsequently combined across segmentation levels.
4 REGIONAL SALIENCY FEATURES
The method represents each region with a 93-dimensional vector combining contrast, backgroundness, and generic regional properties. These descriptors capture visual differences, image-context cues, and appearance or geometry for learned saliency estimation.
- Regional feature vector: Each region is represented by a 93-dimensional feature vector combining regional contrast, regional backgroundness, and generic regional properties.The feature vector is composed of 2 × 29 descriptor dimensions plus 35 generic property dimensions.
- Regional contrast descriptor: Regional contrast uses color and texture features to represent differences between a region and other regions.Color features cover RGB, HSV, and L*a*b* spaces; texture features use LBP and LM filter-bank responses.
- Regional contrast descriptor: The contrast descriptor preserves channel-wise differences instead of directly collapsing them into a single heuristic saliency value.This produces a 29-dimensional regional contrast vector.
- Regional backgroundness descriptor: Backgroundness compares each region with a 15-pixel-wide image border used as a pseudo-background reference.A survey on 5,000 MSRA-B images found that 98% of border pixels belonged to the background.
- Regional property descriptor: Generic regional properties describe appearance distributions and geometry, including a region’s size and position.These properties characterize common color and texture patterns as well as spatial structure.
5 LEARNING
The learning framework generates confident multi-level regions, trains a Random Forest to map regional features to saliency scores, and fuses saliency maps across segmentation levels. It also estimates feature importance and finds that simple averaging performs nearly as well as learned weighted fusion.
- Generating training samples: Training uses multi-level segmentation to generate confident regional samples for supervised saliency regression.Regions are retained when salient-object or background pixels exceed 80%; approximately 6% of regions are discarded as unconfident.
- Generating training samples: A boosted decision tree estimates similarity for adjacent region pairs using 222-dimensional features before hierarchical grouping.The pair features include regional saliency, feature contrast, and boundary geometry.
- Generating training samples: Multi-level segmentations are produced by merging neighboring superpixels in decreasing learned-similarity order while varying the small-region tolerance.This produces multiple segmentation levels for gathering training samples.
- Generating training samples: The learned similarity is hard to generalize across datasets, motivating unsupervised multi-level segmentation during testing.The authors note that an earlier version consequently did not perform best on SED2; the testing procedure is also more efficient.
- Learning the regional saliency regressor: A Random Forest regressor maps each region’s feature vector to a saliency score using split thresholds and leaf predictions averaged across trees.The forest is trained from regional feature vectors and saliency scores; feature importance is estimated during training.
- Learning the multi-level saliency fusor: The final saliency map fuses maps from multiple segmentation levels with a learned linear combination.In practice, averaging the multi-level maps performs nearly as well as a weighted average.
6 EXPERIMENTS
Experiments analyze parameter choices, regional feature importance, benchmark performance, and robustness on challenging salient-object datasets. DRFI consistently outperforms competing methods, while multi-level fusion and selected feature subsets improve or preserve performance.
- Feature importance: Regional contrast is least important on most benchmark datasets, while backgroundness is dataset-dependent and can hurt performance on ECSSD.
- Quantitative comparison: The single-level DRFIs variant performs best on iCoSeg, ECSSD, and DUT-OMRON, while remaining among the top three models on MSRA-B and SED2.
- Quantitative comparison: Multi-level enhancement further improves performance by 1.22% on MSRA-B and 1.78% on DUT-OMRON.
- Robustness analysis: On DUT-OMRON images with salient objects touching borders and far from the center, DRFI remains superior despite declining performance across all methods.
7 DISCUSSIONS AND FUTURE WORK
The paper favors supervised, data-driven saliency integration over fragile heuristic rules, while acknowledging failure modes and several directions for extending DRFI.
- Unsupervised vs Supervised: Heuristic saliency integration rules may generalize poorly because different predefined rules favor different dataset settings.
- Unsupervised vs Supervised: Nearly two million training samples help DRFI perform almost best across six benchmark datasets, including challenging cases unlike its training set.
- Unsupervised vs Supervised: Learning-based saliency detection can run as fast as some heuristic methods once the classifier has been trained.
- Limitations of Our Approach: DRFI may fail on cluttered scenes, especially when background texture is distinctive or salient objects resemble or occupy much of the background.
- Future Work: Future work includes adding saliency cues, improving map fusion, and incorporating depth, temporal consistency, or co-saliency information.