Source-linked AI summary
How hard can it be? Estimating the difficulty of visual search in an image
Radu Tudor Ionescu, Bogdan Alexe, Marius Leordeanu, Marius Popescu, Dim P. Papadopoulos, Vittorio Ferrari
TL;DR
The paper asks how to estimate human visual search difficulty when no suitable image-difficulty dataset exists. It collects response-time annotations, analyzes interpretable properties, and trains a deep-feature regression model whose scores support object-recognition tasks.
Problem
The study addresses estimating image difficulty as human response time for visual search, a problem with little prior computer-vision data.
Method
The authors collect difficulty annotations for PASCAL VOC images, analyze interpretable properties, and train regression models using CNN features.
Results
Difficulty scores improve weakly supervised object localization by 8% and semi-supervised object classification by 1%.
Takeaways & Limitations
Predicted visual search difficulty is useful beyond ranking images, supporting improvements in two object-recognition applications.
Takeaways & Limitations
Human-interpretable image properties analyzed for difficulty are unavailable at test time.
Abstract
from arXiv · showhide
We address the problem of estimating image difficulty defined as the human response time for solving a visual search task. We collect human annotations of image difficulty for the PASCAL VOC 2012 data set through a crowd-sourcing platform. We then analyze what human interpretable image properties can have an impact on visual search difficulty, and how accurate are those properties for predicting difficulty. Next, we build a regression model based on deep features learned with state of the art convolutional neural networks and show better results for predicting the ground-truth visual search difficulty scores produced by human annotators. Our model is able to correctly rank about 75% image pairs according to their difficulty score. We also show that our difficulty predictor generalizes well to new classes not seen during training. Finally, we demonstrate that our predicted difficulty scores are useful for weakly supervised object localization (8% improvement) and semi-supervised object classification (1% improvement).
1. Introduction
The paper estimates visual search difficulty as a human-centered property of images, addressing a sparsely explored problem without an existing difficulty dataset. It collects human annotations, analyzes interpretable image properties, and demonstrates applications in object recognition.
- Visual search difficulty varies across images because some objects are found quickly while others require more intensive human visual processing.
- The study collects response-time-based difficulty annotations for over 10K PASCAL VOC 2012 images and investigates which image properties correlate with difficulty.
- Difficulty scores improve weakly supervised object localization by 8% and semi-supervised object classification by 1%.
- Unlike prior work focused on object-window ranking or segmentation performance, this approach targets higher-level difficulty for general visual search.
2. Image difficulty from a human perspective
The paper operationalizes image difficulty as human response time in object-presence searches and measures it across PASCAL VOC images. Human rankings are consistent, while clutter, object arrangement, size, and class characteristics help explain difficulty.
- Image difficulty is quantified as the time a human needs to decide whether a specified object class is present.
- Annotations cover all 11,540 PASCAL VOC 2012 training and validation images using positive and negative object-presence questions.
- A mean Kendall’s τ of 0.562 among 58 trusted annotators indicates that human rankings agree on about 80% of image pairs.
- The listed image properties are used for analysis and prediction but are unavailable at test time.
- Images become harder when many instances of different classes are scattered across the scene, while smaller mean object area is associated with greater difficulty.
- Difficulty also relates to truncated, occluded, or difficult objects and varies by class, with potted plants, chairs, dining tables, and tv monitors among the hardest.
3. Learning to predict visual search difficulty
The paper predicts visual search difficulty from image features, comparing hand-designed baselines with CNN-based regression models. Deep features with ν-SVR substantially outperform the baselines, reaching about 75% correct ranking of image pairs.
- Our regression model: The proposed model combines CNN features from VGG-f and VGG-verydeep-16 with pyramid representations and ν-SVR regression.The experiments also compare standard, pyramid, and horizontally flipped features with KRR and ν-SVR.
- Baselines: The authors compare random, image area, file size, objectness, edge, and segmentation baselines for predicting human visual search difficulty.These baselines use features computable at test time without manual annotations.
- Baselines: Combined baseline features with ν-SVR achieve Kendall’s τ≈0.30, correctly ranking about 65% of image pairs.Random scores have almost zero accuracy, while mid-level features perform better but remain limited.
- Experimental analysis: The best model reaches Kendall’s τ=0.472 and correctly ranks about 75% of image pairs, outperforming the baselines and image-property predictors.Its performance exceeds approaches requiring object counts, classes, or bounding boxes unavailable at test time.
- Experimental analysis: Predicted and ground-truth difficulty scores show strong correlation, with the regression line oriented almost diagonally.The model ranks simple images with few objects and uniform backgrounds below cluttered images with many objects and complex backgrounds.
- Experimental analysis: The model’s higher-level CNN features outperform lower-level edge, objectness, and segmentation features for estimating human visual-search difficulty.The authors relate this result to the task’s dependence on object detection and recognition.
4. Applications
The paper applies its image-difficulty measure to weakly supervised object localization and semi-supervised object classification, using difficulty to order or select training examples. These applications improve localization and classification performance relative to baseline strategies.
- The difficulty measure is used in two applications: weakly supervised object localization and semi-supervised object classification.
- Weakly supervised object localization: In WSOL, Easy-to-Hard MIL ranks images by estimated difficulty before passing them to standard MIL.The protocol integrates difficulty as an additional cue in the learning process.
- Weakly supervised object localization: +7% after the second batch and +8.4% after the third demonstrate that processing order matters in WSOL.Processing easier images initially results in better class models that improve later stages.
- Semi-supervised object classification: Figure 4 compares mAP as training-set size grows when automatically labeled samples are added using different heuristics against BASIC.
- Semi-supervised object classification: In semi-supervised classification, predicted difficulty scores achieve 88.4% ± 0.6% mAP versus 87.8% ± 0.6% for BASIC.Ground-truth difficulty scores achieve 88.5% ± 0.7%, while the reported improvement remains below 1%.
5. Future work
The authors propose using the difficulty measure in curriculum learning to optimize training of deep learning models for vision tasks.
- The difficulty measure is proposed for curriculum learning settings to optimize CNN training across various vision tasks.