PaperlayerView exact revision on arXiv

Source-linked AI summary

MIMONet: Multi-scale Input and Multi-scale Output Network for Salient Object Detection

Zhaojian Yao, Wei Gao, Tiesong Zhao, Hui Yuan, Sam Kwong

arXiv:2608.25733v1cs.CV

TL;DR

Single-scale saliency models struggle to learn salient-object size variation, although recognizing objects of different sizes is important. MIMONet processes three resolutions in parallel, exchanges multi-scale features, enhances them with MSP, and jointly supervises saliency maps with JSL. Experiments report stronger detection capability and better evaluation scores across multiple datasets than existing models.

  • Problem

    Single-size inputs make multi-level features struggle to learn salient-object size variation, limiting fully explored object-scale variation learning for multi-scale detection.

  • Method

    MIMONet processes three differently resolved images through interacting encoder branches, uses MSP for multi-scale structural features, and applies JSL to align outputs and preserve boundaries.

  • Results

    MIMONet achieves better evaluation scores than competing methods in most cases across multiple datasets, while remaining under 30M parameters and retaining real-time inference speed.

  • Takeaways & Limitations

    Multi-scale inputs, feature exchange, MSP, and joint output supervision enable MIMONet to identify objects with different sizes and produce clearer salient regions.

Abstract

from arXiv · show

The existing methods for saliency detection task focus on the application of multi-level features, aiming to take advantage of the respective strengths of high- and low-level features. However, because the inputs of these models are single-size images, their multi-level features have difficulty in learning the knowledge of size variations of salient objects. Object-scale variation learning has great potential for detecting multi-scale objects, which has not been fully explored by existing methods. To improve the recognition ability of a model for objects with different sizes, we are inspired by the image pyramid to propose a Multi-scale Input and Multi-scale Output Network (MIMONet). In MIMONet, we extract multi-level features for three images with different resolutions to form three encoder branches, and information will be exchanged between the branches. The advantage of this approach is that the features of one branch can learn the knowledge of target size variation from the features of the other two branches. In addition, we design a Multi-scale Perception (MSP) module, in which the input feature layer is divided into several sub-layers with different resolutions. Capturing the multi-level structure information of the objects in these sub-layers can make the objects more fully perceived. For network training, we propose a Joint Saliency Loss (JSL), which can constrain multiple saliency maps output by the network to identify the same foreground objects, and induce their boundaries to be preserved clearly. Experimental results show that MIMONet has stronger detection capabilities and harvests better evaluation scores on multiple datasets compared to existing models. The code of our model will be released.

I. INTRODUCTION

MIMONet addresses salient-object size variation by processing multiple image resolutions in parallel and exchanging multi-scale features. It further enhances multi-scale perception and jointly supervises outputs so they predict consistent salient regions with clear boundaries.

  • Motivation: Single-scale RGB saliency models combine high-level semantic and low-level detail features but underuse information about salient-object size variation.Salient objects vary across scenarios and within scenes, making accurate localization challenging.
  • MIMONet: MIMONet feeds three differently resolved images into parallel encoder branches and exchanges same-level multi-scale features between them.Shared backbone parameters allow branches to complement one another while learning cues about object-size changes.
  • MIMONet: Multi-scale feature exchange combines semantic, detail, and object-size variation cues to improve detection of salient objects with variable sizes.The exchanged features are fused during decoding to support foreground prediction for each input resolution.
  • MSP module: The MSP module uses three scale-transformed branches to extract multi-level structural information from feature layers and enhance perception of targets with various sizes.Its branches process differently resolved sub-layers before aggregating their results.
  • Joint saliency loss: JSL jointly constrains multiple saliency maps to identify the same salient regions while preserving object boundaries.The loss addresses the independence of saliency-map supervision in existing methods.

II. RELATED WORK

Prior saliency methods mainly fuse multi-level features from a single-scale image and independently supervise multiple maps. MIMONet instead introduces multi-size information during encoding, refines features through MSP, and imposes commonality constraints across outputs.

  • Multi-level feature fusion: Existing saliency models commonly fuse high- and low-level features from a single-scale image to combine semantic and detail information.These approaches use complementary multi-level features but do not sufficiently capture target-size information.
  • Multi-level feature fusion: Single-scale feature fusion gives insufficient attention to multi-size target information, reducing model flexibility when salient-object size changes.MIMONet addresses this by introducing multi-size object information during encoding and constraining each size-specific output during decoding.
  • Network supervision: Existing multi-output supervision optimizes saliency maps independently, so predictions may fail to emphasize the same complete salient regions.MIMONet adds a commonality constraint to encourage joint prediction of shared salient regions.
  • Multi-scale Perception Module: MSP splits a feature layer into three branches that process low-, raw-, and high-resolution features before aggregation.The branches use different scale transformations and convolution kernels to extract structural information at multiple levels.
  • Multi-scale Perception Module: Aggregating MSP branch outputs produces enhanced features that highlight complete salient regions more than the original feature layer.The module is described as refining internal feature information for more accurate object representation.

B. Multi-scale Input and Multi-scale Output Network

MIMONet combines three resolution inputs, cross-branch scale-information exchange, multi-level feature fusion, and MSP enhancement to detect salient objects across size changes. Each branch produces a saliency map from optimized features.

  • Illustrated pipeline: Figures 6–8 depict the inference pipeline, multiple predictions from multi-scale inputs, and common-boundary generation from saliency maps.The supplied figure descriptions identify the main processing stages and the boundary-construction operation.
  • Multi-scale inputs: MIMONet resizes an RGB image to low-, original-, and high-resolution inputs, processed simultaneously by a shared ResNet50 backbone.The scaling factors are 0.5 and 1.5 for the low- and high-resolution images.
  • Feature exchange: Same-level features exchange information across the three branches so each feature layer can acquire cues about object-size changes.The exchanged features are then integrated with multi-level features to form complementary representations.
  • Multi-scale perception: The fused complementary features are processed by MSP modules, providing a second scale transformation that more fully captures object-size variation.The resulting Klow, Kmid, and Khigh features are enhanced before saliency prediction.
  • Multi-scale outputs: Each enhanced branch feature predicts a corresponding saliency map, producing multi-scale outputs for the three input resolutions.The framework is designed to identify salient objects whose sizes vary across scenes.

C. Loss Function

The loss function combines independent supervision of three saliency outputs with joint constraints that align their predicted regions and preserve object boundaries.

  • BCE and IoU losses constrain each saliency map against its label at pixel-level and whole-region scales.
  • MIMONet independently supervises saliency maps produced at three different resolutions.
  • The joint saliency loss forms a common saliency map by combining the low-, middle-, and high-resolution predictions.
  • Parallel branches generate foreground-background separation maps from the common saliency map using dynamically varied processing structures.
  • The separation maps are fused into a boundary map, which is directly constrained against the boundary label.
  • JSL jointly encourages the three predictions to identify common salient regions while retaining their boundaries.

1) Evaluation Datasets:

Experiments use six established salient-object-detection benchmarks covering complex scenes, varied object structures, multiple objects, natural scenarios, and diverse object attributes.

  • The evaluation uses ECSSD, PASCAL-S, HKU-IS, DUT-OMRON, DUTS, and SOC as benchmark datasets.Their sizes range from 850 to 10,533 training images, with DUTS also containing 5,019 test images and SOC containing 1,200 test images.

2) Evaluation Metrics:

Model performance is assessed with complementary metrics covering precision-recall balance, pixel-wise error, structural similarity, and enhanced alignment.

  • F-measure balances precision and recall, with maximum, average, and weighted variants reported.
  • Mean Absolute Error measures the pixel-wise average difference between the predicted saliency map and ground truth.
  • S-measure and E-measure are also used to evaluate saliency-detection performance.

3) Implementation Details:

The implementation uses a ResNet50-based PyTorch model trained on DUTS with 352 × 352 inputs and specified loss-balance parameters.

  • The model is implemented in PyTorch and uses ResNet50 to initialize the backbone’s four feature layers.
  • Training uses the DUTS training set with original images resized to 352 × 352.
  • All non-backbone 3 × 3 convolutions are followed by group normalization and PReLU.
  • The experiments use μR = 3 and μB = 0.1, and run on one NVIDIA TESLA V-100 GPU.

B. Comparison with State-of-the-arts

MIMONet is evaluated against 21 recent state-of-the-art methods and achieves strong quantitative and qualitative performance across multiple datasets. Its multi-scale design improves handling of salient objects with varying sizes while retaining relatively low complexity and real-time inference.

  • Quantitative Evaluation: MIMONet achieves better evaluation scores than competing methods in most cases and remains close to the best performance otherwise.It has the lowest M scores on most datasets, indicating greater similarity between saliency maps and ground truths.
  • Quantitative Evaluation: With fewer than 30M parameters, MIMONet improves performance without relying on a larger model.Despite processing three resolutions simultaneously, it maintains real-time inference speed.
  • Qualitative Evaluation: MIMONet produces clearer saliency maps and stronger background suppression across small, medium, and large salient objects.The qualitative comparisons also report better results in low-contrast, complex, and cluttered scenes.
  • Effectiveness of MIMONet: Adding different-resolution inputs improves detection when N > 1, while the three-input configuration learns object-size variation more effectively than two-input settings.The N = 5 configuration performs worse than MIMONet in some cases because additional parameters increase optimization difficulty.
  • Effectiveness of MIMONet: A moderate scaling-factor range improves inference, whereas a wider range weakens performance because unstable scale changes hinder accurate variation-pattern learning.The passage identifies scaling factors as a key factor affecting performance.
  • Effectiveness of MIMONet: Independent backbones increase network parameters and complexity while reducing performance compared with shared-backbone designs.The comparison supports sharing backbone parameters across branches.

2) Importance of MSP:

The Multi-scale Perception module uses hierarchical scale transformations to make feature layers perceive object structures at multiple resolutions. Experiments show that these transformations and the MSP design contribute to stronger salient-region detection.

  • Importance of MSP: MSP divides an input feature layer into three differently resolved sub-layers that jointly learn low-, middle-, and high-scale object structures.This hierarchical representation is intended to strengthen feature expression for targets of different sizes.
  • Importance of MSP: Removing scale transformation causes poorer detection performance because the model loses simulated scale variation for perceiving small and large regions.The transformations consist of up-sampling and down-sampling operations.
  • Importance of MSP: Unlike PPM, which mainly fuses down-sampled features and can lose detail, MSP preserves multi-scale structure information relevant to region boundaries.The comparison identifies detail loss as detrimental to boundary recovery in saliency detection.
  • Importance of MSP: MSP plays an important role in MIMONet’s detection of salient regions.The cited experiment reports this conclusion for the module comparison.

3) Contribution of JSL:

JSL jointly constrains saliency predictions to identify common foreground regions and preserves their boundaries through boundary-aware loss calculation. This couples multi-resolution outputs around shared salient content.

  • Contribution of JSL: JSL constrains multiple saliency maps to predict common salient regions before generating a common boundary map.The common boundary map is compared directly with the boundary label.
  • Contribution of JSL: Boundary-loss calculation in JSL helps preserve region boundaries while enforcing agreement among multi-resolution predictions.The conclusion states that JSL makes region boundaries more clearly preserved.
  • Contribution of JSL: Multi-scale outputs provide supervision that helps the network extract precise common saliency cues from multi-scale images.This supervision complements multi-scale input by making predictions focus on the same salient regions.
Loading 2608.25733v1…