Source-linked AI summary
EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction
Sen Jia, Neil D. B. Bruce
TL;DR
Saliency prediction needs richer visual knowledge, but combining deeper or cross-domain CNN models can make systems complex, space-intensive, and difficult to train. EML-NET uses separately trained encoder and decoder modules, with multiple CNNs in the encoder, and achieves state-of-the-art results on SALICON, MIT300, and CAT2000. Its main supported boundary is that CAT2000 categories unlike common natural-image datasets remain difficult for the model.
Problem
Saliency systems need richer visual information than the 1,000 ImageNet object categories provide, but combining models from different domains increases complexity, objective-balancing demands, and computational space.
Method
EML-NET separately trains the encoder, decoder, and each encoder CNN, allowing multiple deep models pretrained on different datasets to be combined for saliency prediction.
Results
EML-NET achieves state-of-the-art results on the public SALICON, MIT300, and CAT2000 saliency benchmarks and reaches first place on the SALICON-2017 leaderboard by NSS.
Takeaways & Limitations
The scalable design supports more efficient use of deeper CNNs and expansion with prior knowledge from additional vision tasks.
Takeaways & Limitations
On CAT2000, EML-NET does not achieve higher performance, with NSS matching the cited prior method because several categories differ substantially from natural images in ImageNet or PLACE365.
Abstract
from arXiv · showhide
Saliency prediction can benefit from training that involves scene understanding that may be tangential to the central task; this may include understanding places, spatial layout, objects or involve different datasets and their bias. One can combine models, but to do this in a sophisticated manner can be complex, and also result in unwieldy networks or produce competing objectives that are hard to balance. In this paper, we propose a scalable system to leverage multiple powerful deep CNN models to better extract visual features for saliency prediction. Our design differs from previous studies in that the whole system is trained in an almost end-to-end piece-wise fashion. The encoder and decoder components are separately trained to deal with complexity tied to the computational paradigm and required space. Furthermore, the encoder can contain more than one CNN model to extract features, and models can have different architectures or be pre-trained on different datasets. This parallel design yields a better computational paradigm overcoming limits to the variety of information or inference that can be combined at the encoder stage towards deeper networks and a more powerful encoding. Our network can be easily expanded almost without any additional cost, and other pre-trained CNN models can be incorporated availing a wider range of visual knowledge. We denote our expandable multi-layer network as EML-NET and our method achieves the state-of-the-art results on the public saliency benchmarks, SALICON, MIT300 and CAT2000.
1. Introduction
Saliency prediction uses visual features to model where humans look, but deeper and multi-source CNN representations create training and computational challenges. EML-NET addresses these challenges with a modular, expandable encoder-decoder system and a combined loss across saliency metrics.
- Motivation: Saliency prediction models where humans fixate on regions of interest while less salient image areas are ignored.The human visual system’s attention mechanism supports multiple vision tasks, including object classification, video analysis, image compression, action classification, and quality assessment.
- Motivation: Existing saliency systems commonly use relatively shallow ImageNet-pretrained CNNs, while deeper models can require larger training sets and more computational space.The SALICON dataset contains 10,000 annotated training images, addressing the limited-data concern described for deeper CNNs.
- Motivation: ImageNet provides limited object information through 1,000 categories, motivating the use of visual knowledge from other domains such as face detection or multiple vision tasks.Combining CNNs pretrained for different tasks can increase complexity, objective-balancing demands, and space requirements.
- Proposed approach: EML-NET separately trains the encoder, decoder, and each encoder CNN, reducing computational complexity and space requirements while enabling deeper CNNs and multiple pretrained models.The decoder must be retrained when a new encoder model is added, but the paper describes that retraining cost as very small.
- Training and evaluation: EML-NET combines modified CC and NSS dissimilarities with KLD in a loss function to satisfy multiple competing saliency objectives together.The method is trained on SALICON and evaluated on SALICON, MIT300, and CAT2000; the authors report outperforming other state-of-the-art algorithms, especially on NSS.
2. Related Work
Prior saliency work progressed from hand-crafted visual features to CNN-based representations and recurrent or scene-recognition components. EML-NET instead uses separately trained, scalable CNN modules to combine deeper and cross-domain features.
- Early saliency methods: Early saliency methods extracted hand-crafted colour, intensity, orientation, edge, or independent-component features to construct saliency maps.These approaches include 42 feature maps and image-patch analysis based on independent component analysis.
- CNN-based methods: CNNs enabled discriminative feature extraction for saliency, including combining multiple layers and transferring features learned for object recognition.This work established the use of off-the-shelf CNN features for predicting saliency maps.
- Recent methods: Later systems combined ResNet-50 with recurrent networks and used multiple saliency metrics to achieve state-of-the-art results on MIT300.The cited approach also modified NSS and CC into dissimilarity terms for a combined loss.
- Limitations of prior designs: Scene-recognition features were incorporated in prior work, but joint training made it difficult to extend the system with additional models or inference modes.That design also required resizing the input image for the scene-recognition model because of its linear layer.
- EML-NET: EML-NET uses NasNet pretrained on ImageNet and DenseNet pretrained on PLACE365 in an almost end-to-end design, while remaining potentially compatible with recurrent or task-specific models.The paper presents it as an alternative to a simple ResNet-50 encoder paired with a complex recurrent decoder.
3. Expandable Saliency System
EML-NET uses separately trained CNN encoders and a decoder to combine deep, multi-layer features while controlling computational space. Its training loss combines modified CC and NSS with KLD to address differing saliency objectives.
- System overview: EML-NET maps images to encoder features and uses a decoder to up-sample them into predicted saliency maps.The encoder applies CNN layers, while the decoder predicts saliency from extracted features.
- Encoding Stage: DenseNet-161 and NasNet-Large provide features learned from different visual domains, with PLACE365 pre-training for DenseNet and ImageNet pre-training for NasNet.The design combines prior knowledge from multiple vision tasks and datasets.
- System overview: The system trains the encoder, decoder, and each encoder CNN separately, reducing computational complexity and required space.This piecewise strategy enables deeper CNN models such as DenseNet and NasNet.
- Decoding Stage: The decoder combines seven compressed feature maps—four from DenseNet and three from NasNet—after aligning them to the largest intermediate-map size.Each selected layer is compressed with conv1@1 and ReLU; alignment uses bilinear up-sampling to 160×120 for 640×480 inputs.
- Decoding Stage: The decoder has 13,543 parameters, reported as 18 hundred times fewer than ResNet-50 on ImageNet images, while total space depends on the largest single CNN.This scaling property supports expansion with additional CNN models.
- Combined Loss Function: The training loss sums KLD with modified CC and NSS terms, while original metrics are retained for benchmark reporting.The combined objective is Loss = NSS′+CC′+KLD, and the metrics emphasize different prediction attributes.
4. Experiments
Experiments evaluate EML-NET across SALICON, MIT300, and CAT2000, including architecture comparisons and public-benchmark results. EML-NET performs strongly on SALICON and MIT300 but does not improve over prior performance on CAT2000’s uncommon image categories.
- Datasets: SALICON provides 10,000 training images, while MIT300 uses eye-tracking labels from 39 observers and CAT2000 includes 2,000 training images plus 2,000 test images.MIT1003 is used to fine-tune EML-NET for MIT300, and CAT2000 training images are split into 1,800 fine-tuning and 200 validation images.
- Architecture comparison: NasNet outperforms DenseNet on all reported SALICON validation metrics, while combining both models improves some measures but not the AUC metrics.The authors attribute the differing AUC behavior to metric bias and differences in the regions emphasized by the models.
- Public benchmarks: EML-NET achieves first place on the SALICON-2017 test leaderboard and records the highest NSS score.The evaluation uses 5,000 submitted saliency maps and multiple saliency metrics.
- Public benchmarks: On MIT300, EML-NET achieves the highest NSS score of 2.47 and the lowest earth mover distance of 1.84, 0.2 below the second-place result.The model is fine-tuned on MIT1003 before evaluation on MIT300.
- Public benchmarks: On CAT2000, EML-NET matches the NSS score of a comparison method rather than achieving higher performance.The authors associate low performance with rare categories such as Satellite, Art, Cartoon, Inverted, and Jumbled, where ImageNet or PLACE365 prior knowledge is less applicable.
5. Conclusion
EML-NET shows that deeper CNNs can improve saliency features when training data is sufficiently large, while its scalable design supports combining networks of varying depth and complexity. The method achieves state-of-the-art results on public saliency benchmarks.
- Deeper CNN models deliver better saliency features when the training set is large enough.
- EML-NET scales to networks of arbitrary depth and complexity that can encode complementary information for visual saliency.
- The method achieves state-of-the-art results on public saliency benchmarks.