Source-linked AI summary
Label Decoupling Framework for Salient Object Detection
Jun Wei, Shuhui Wang, Zhe Wu, Chi Su, Qingming Huang, Qi Tian
TL;DR
Saliency prediction is especially difficult near object edges, where imbalanced pixels make direct edge supervision suboptimal. LDF decouples saliency labels into body and detail maps and uses an interacting two-branch network for iterative refinement. Across six benchmark datasets, it outperforms state-of-the-art methods under different evaluation metrics.
Problem
Edge-adjacent pixels are harder to predict than distant pixels, while edge supervision is limited by imbalanced edge-pixel distributions and overlooks nearby hard pixels.
Method
LDF decomposes saliency labels into body and detail maps, then uses two FIN branches with feature interaction to fuse and iteratively refine their predictions.
Results
LDF outperforms state-of-the-art salient object detection methods on six datasets under different evaluation metrics.
Takeaways & Limitations
Detail maps provide broader supervision around edges, while body maps reduce edge-related distraction and support complementary branch learning.
Abstract
from arXiv · showhide
To get more accurate saliency maps, recent methods mainly focus on aggregating multi-level features from fully convolutional network (FCN) and introducing edge information as auxiliary supervision. Though remarkable progress has been achieved, we observe that the closer the pixel is to the edge, the more difficult it is to be predicted, because edge pixels have a very imbalance distribution. To address this problem, we propose a label decoupling framework (LDF) which consists of a label decoupling (LD) procedure and a feature interaction network (FIN). LD explicitly decomposes the original saliency map into body map and detail map, where body map concentrates on center areas of objects and detail map focuses on regions around edges. Detail map works better because it involves much more pixels than traditional edge supervision. Different from saliency map, body map discards edge pixels and only pays attention to center areas. This successfully avoids the distraction from edge pixels during training. Therefore, we employ two branches in FIN to deal with body map and detail map respectively. Feature interaction (FI) is designed to fuse the two complementary branches to predict the saliency map, which is then used to refine the two branches again. This iterative refinement is helpful for learning better representations and more precise saliency maps. Comprehensive experiments on six benchmark datasets demonstrate that LDF outperforms state-of-the-art approaches on different evaluation metrics.
1. Introduction
The paper identifies edge-adjacent pixels as especially difficult for saliency prediction and proposes LDF to decouple labels into complementary body and detail maps. Its feature interaction network iteratively combines these branches, and experiments on six datasets report substantial improvements over state-of-the-art models.
- Motivation: Edge pixels and nearby pixels have larger prediction errors than pixels farther from object boundaries.MAEedge is much larger than MAEglobal, and error rises sharply when normalized edge distance falls below 0.4.
- Motivation: Traditional edge supervision overlooks many difficult pixels near boundaries and suffers from imbalanced edge-pixel distributions.The paper notes that using only edge pixels fails to fully use hard pixels close to edges.
- Proposed Framework: Label decoupling decomposes each saliency label into a body map for central regions and a detail map covering edges and nearby pixels.The detail map includes more pixels than a pure edge map, while the body map avoids interference from edge-adjacent pixels.
- Proposed Framework: FIN uses two complementary branches and iteratively exchanges information to refine branch representations and saliency predictions.The branches are supervised separately and fused to predict the saliency map before refinement.
- Evaluation: Experiments on six SOD datasets show that LDF outperforms state-of-the-art models by a large margin.The study also evaluates LDF across challenging scenes in the SOC dataset.
2. Related Work
Related SOD methods primarily aggregate multi-scale features or add edge labels to improve saliency maps. LDF instead decouples labels into body and detail maps while using feature interaction to exploit their complementarity.
- Background: Traditional SOD methods rely on intrinsic cues such as color and texture but lack high-level semantic representation in complex scenarios.Deep learning models subsequently improved feature representation and robustness.
- Aggregation-based Models: Aggregation-based models commonly use encoder-decoder architectures to extract multi-scale features and integrate context across levels.Representative designs include shortcut connections, reverse attention, contextual attention, and bidirectional message passing.
- Feature Interaction: The proposed framework uses feature interaction to iteratively exchange complementary information between the body and detail branches.The framework overview includes separate encoders and decoders for feature extraction, interaction, body prediction, and detail prediction.
- Edge-based Models: Edge-based models add edge labels or edge losses to assist saliency-map generation and boundary refinement.Approaches include binary cross-entropy edge loss, hybrid losses, auxiliary edge data, and bidirectional refinement between segmentation and edge maps.
- Label Decoupling Framework: LDF addresses imbalanced edge distributions by explicitly separating the saliency label into body and detail maps.The detail map supports edge-feature learning, while the body map reduces distraction from pixels near edges.
3. Methodology
The methodology decouples saliency supervision into body and detail labels, then uses complementary feature branches and iterative interaction to generate refined saliency maps.
- 3.1. Label Decoupling: The transformed label becomes a body label emphasizing central, relatively easy pixels and a detail label emphasizing regions near object boundaries.The detail label includes edges and nearby pixels, while multiplying both labels by the original binary image removes background interference.
- 3.2. Feature Extraction: The backbone retains convolutional blocks and produces five progressively downsampled feature scales, while the decoder uses scales 2–5 adapted to body and detail prediction.The resulting body and detail feature groups are squeezed to 64 channels before being sent to the decoder network.
- 3.3. Feature Interaction Network: FIN uses two branches for body and detail supervision and exchanges information between their complementary features through an interaction encoder.The interaction decoder processes concatenated branch features and fuses interaction features with backbone features to produce more accurate saliency maps.
- 3.4. Loss Function: The training loss sums losses across all iterations, with each iteration combining body, detail, and segmentation losses.Iteration weights are set to α_k = 1, and binary cross entropy is used for body and detail losses; IoU loss is used for segmentation because it measures whole-image similarity.
- 3.4. Loss Function: IoU loss is not applied to body and detail labels because those labels are nonbinary, whereas IoU loss requires binary ground truth.
4. Experiments
Experiments evaluate LDF on six benchmark datasets using standard saliency metrics, comparisons, visualizations, and ablations. LDF outperforms prior methods, improves edge-region accuracy, and benefits from label decoupling and feature interaction.
- Datasets and Evaluation Metrics: LDF is evaluated on six benchmark datasets using MAE, mean F-measure, E-measure, weighted F-measure, S-measure, and PR/F-measure curves.DUTS-TR provides training data, while the other datasets are used for evaluation.
- Ablation Studies: One feature interaction performs better than no interaction, while additional interactions reduce performance because they deepen the network and complicate optimization.The authors select one interaction to balance optimization and performance.
- Ablation Studies: Supervision combinations using detail labels outperform those using edge labels, while body labels outperform saliency labels.The authors attribute the body-label result to reduced interference from edge pixels during representation learning.
- Comparison with State-of-the-arts: LDF outperforms 14 state-of-the-art methods by a large margin on MAE, mean F-measure, and E-measure across six datasets.The compared methods include EGNet-R, SCRN, SIBA, PoolNet, and other published SOD models.
- Comparison with State-of-the-arts: LDF’s precision-recall and F-measure curves consistently lie above competing methods on five datasets.These curves summarize performance across thresholds.
- Comparison with State-of-the-arts: LDF achieves the minimum error across pixel distances, especially near edges, and produces clearer maps with better background suppression in challenging scenes.Visual comparisons cover cluttered backgrounds, manufactured structures, and low-contrast foregrounds.
- Comparison with State-of-the-arts: On the SOC dataset, LDF achieves the best results for most attributes except “BO,” indicating good generalization across challenging scenes.The evaluation reports mean F-measure scores for nine attributes and an overall score.
5. Conclusion
The proposed label decoupling framework separates saliency supervision into body and detail maps, then uses feature interaction to exploit their complementarity. Experiments across six datasets show improved performance over direct saliency-map supervision and state-of-the-art methods.
- The framework decouples saliency labels into body and detail maps, with detail maps learning edge features and body maps avoiding distraction from nearby-edge pixels.The body map focuses away from edges, while the detail map includes edges and surrounding pixels.
- Feature interaction combines the complementary body and detail branches to make full use of both representations.
- The proposed method achieves better performance than direct supervision with saliency maps.
- Experiments on six datasets show that the framework outperforms state-of-the-art methods under different evaluation metrics.