Source-linked AI summary
Black-box Explanation of Object Detectors via Saliency Maps
Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad I. Morariu, Ashutosh Mehra, Vicente Ordonez, Kate Saenko
TL;DR
Object detectors are difficult to explain because existing saliency methods do not directly handle localization alongside classification. D-RISE uses black-box randomized masking and a detection similarity metric to generate explanations, demonstrating applicability across YOLOv3 and Faster R-CNN while revealing reliance on context and discriminative object regions.
Problem
Existing saliency methods are difficult to apply to object detectors because explanations must cover localization, classification, and multiple detection proposals.
Method
D-RISE is a black-box attribution method that uses randomized input masking and detection similarity to explain localization and classification for arbitrary object detectors.
Results
D-RISE explains both YOLOv3 and Faster R-CNN and reveals detectors’ use of contextual regions and discriminative object parts.
Takeaways & Limitations
The explanations support error analysis and provide model developers with insights into detector behavior and pathological biases.
Takeaways & Limitations
Saliency evaluations based on ground-truth masks assume predictions rely only on objects, an assumption that fails for models using important context.
Abstract
from arXiv · showhide
We propose D-RISE, a method for generating visual explanations for the predictions of object detectors. Utilizing the proposed similarity metric that accounts for both localization and categorization aspects of object detection allows our method to produce saliency maps that show image areas that most affect the prediction. D-RISE can be considered "black-box" in the software testing sense, as it only needs access to the inputs and outputs of an object detector. Compared to gradient-based methods, D-RISE is more general and agnostic to the particular type of object detector being tested, and does not need knowledge of the inner workings of the model. We show that D-RISE can be easily applied to different object detectors including one-stage detectors such as YOLOv3 and two-stage detectors such as Faster-RCNN. We present a detailed analysis of the generated visual explanations to highlight the utilization of context and possible biases learned by object detectors.
1. Introduction
D-RISE addresses the underexplored problem of explaining object detectors by producing saliency maps for both detection localization and classification. It is a black-box method that applies across detector architectures and supports analysis of contextual cues, biases, and errors.
- Motivation: Object detectors remain difficult to understand, debug, and improve because deep neural networks make complex, unintuitive decisions.Existing explanation techniques can help identify model biases, support debugging, and increase user trust.
- Analysis and evaluation: D-RISE reveals that detectors may use contextual regions and discriminative object parts, such as ski poles for skis or an Apple logo for laptops.The paper also analyzes potential sources of errors and bias in detectors trained on MS-COCO, using automated metrics, a user study, and synthetic-marker bias discovery.
- Problem: Existing attribution techniques are ill-suited to object detection because detectors require explanations for both bounding-box classification and localization.These methods are often tied to model architecture and rely on assumptions about learned feature maps or activations.
- Method: D-RISE is a black-box attribution technique that uses a detection similarity metric and input masking to explain both localization and classification without gradients or detector internals.The method produces saliency maps by relating target detections to detector outputs from masked images.
- Generality: D-RISE generalizes across one-stage YOLOv3 and two-stage Faster R-CNN object detectors.The paper uses these detectors with different architectural designs to demonstrate generalizability.
2. Related Work
Related work spans detector taxonomies, white-box and perturbation-based saliency methods, and prior detector explainability studies. D-RISE extends random-mask saliency to black-box object detectors, addressing their localization and multiple-proposal outputs.
- Object detector types: Object detectors are categorized as two-stage methods such as Faster R-CNN or one-stage methods such as YOLO, SSD, and CornerNet.Two-stage detectors select sparse regions of interest before classifying candidate regions, whereas one-stage methods use a single network without ROI pooling.
- Detector explainability: Prior detector-explainability work visualized feature spaces, analyzed biases, and applied tailored white-box saliency methods to SSD.These approaches differ from D-RISE, which treats detectors as black boxes.
- Saliency methods: Saliency methods include white-box backpropagation through network layers and perturbation-based approaches that infer importance from output changes after modifying image regions.Examples include Gradients, Excitation Backpropagation, Layer-wise Relevance Propagation, occlusion, noise, inpainting, blurring, and LIME.
- Saliency methods: RISE generates random masks, applies a classifier to masked inputs, and combines the masks using predicted class probabilities as weights to form a saliency map.D-RISE uses this masking technique for object detectors rather than image classifiers.
- Black-box explanation: Black-box methods are typically slower but offer greater generalizability, easier application, and access to proprietary models or APIs unavailable to white-box analysis.They directly measure the effect of input ablations rather than relying on heuristic importance rules.
- Object detector explanation: Object detectors require explanations for both class probabilities and bounding-box locations while producing multiple detection proposals that may differ substantially.Methods designed to explain a scalar output, such as a classification score, therefore do not directly capture all detector outputs.
3. Method
D-RISE generates black-box saliency maps for object-detector predictions by perturbing randomized image regions and measuring changes in detector outputs. Its detection-specific similarity combines localization, class-probability, and, when available, objectness agreement to weight the masks.
- Black-box formulation: D-RISE requires only detector inputs and outputs, without access to the model’s weights, gradients, or architecture.It adapts RISE’s randomized masking idea to object detectors and infers pixel importance from output changes.
- Detection representation: Object-detector explanations must account for multiple proposals encoding localization, class probabilities, and optionally objectness, rather than a single classification output.Final detections are obtained after proposal refinement through confidence thresholding and non-maximum suppression.
- Target flexibility: D-RISE can explain arbitrary detection vectors, including objects missed by the detector, because targets need not be produced by the model.The method constructs target vectors from bounding-box corners, objectness set to 1, and one-hot class probabilities.
- Saliency-map generation: D-RISE samples binary masks, upsamples and crops them, runs the detector on each masked image, matches each target to its highest-similarity proposal, and computes a weighted mask sum.The detector can be run once per masked image to generate saliency maps for all target detections.
- Detection similarity: The similarity between target and proposal detections is the product s(dt, dj) = sL(dt, dj) · sP(dt, dj) · sO(dt, dj), combining bounding-box IoU, class-probability cosine similarity, and objectness similarity when applicable.For the experiments, target detections use Ot = 1, and the multiplicative form makes low agreement in any component reduce total similarity.
- Implementation: With N = 5000 masks, inference takes approximately 70s per image for YOLOv3 and 170s for Faster R-CNN on an NVidia Tesla V100.The reported runtime depends only on the number of masks and covers all detections in the image.
4. Experiments and Results
Experiments on MS-COCO evaluate D-RISE across YOLOv3 and Faster R-CNN against classification-saliency baselines. Results show stronger explanations, support analysis of detector errors and contextual biases, and are preferred by human raters.
- Experimental setup: Experiments use MS-COCO detections from YOLOv3 and Faster R-CNN, with GradCAM and Gradients as classification-saliency baselines.The baselines explain the class-probability score of each detection vector.
- Quantitative evaluation: D-RISE outperforms classification saliency methods on all three reported metrics: Pointing Game, Deletion, and Insertion.The metrics are evaluated on YOLOv3 detections from the MS-COCO 2017 validation split.
- Error analysis: For misclassification and localization errors, comparing saliency maps for predicted and ground-truth detections identifies image regions contributing to class confusion or poor boxes.The method analyzes correctly localized but miscategorized regions and correctly classified but poorly localized regions.
- Error analysis: D-RISE explains missed detections by producing saliency maps for arbitrary detection vectors, revealing discriminative image features even when the model detects no object.This capability is unavailable to gradient-based explanations restricted to model-produced detections.
- Holistic analysis: Average maps show category-specific reliance on object parts and context, including upper-body emphasis for person detection and saliency above sinks associated with unlabeled faucets.Other categories, such as giraffe, show saliency distributed across the whole object.
- Human evaluation: 50.2% vs 27.4% of users found explanations from YOLOv3, the more accurate model, better or more trustworthy than those from the comparison model.The study collected five responses per object from 32 unique Mechanical Turk users.
5. Conclusion
The paper proposes a general saliency-based method for explaining black-box object detectors. It demonstrates the method’s usefulness for error analysis, model development, and identifying pathological biases through per-class average saliency maps.
- 5. Conclusion: The method provides saliency-based explanations for black-box object detectors.It is presented as a novel approach to explain detector behavior.
- 5. Conclusion: The approach is general enough to apply to many different object-detection architectures.
- 5. Conclusion: Per-class average saliency maps support error analysis and provide insights to model developers.
- 5. Conclusion: The method can identify pathological biases in model behavior.
6. Appendix
The appendix documents the human-feedback interface and quantitative evaluation setup for D-RISE, alongside analyses of detector architecture, learned marker biases, and class-level saliency maps. It also reports that stronger-model explanations were more often judged better or more trustworthy than weaker-model explanations.
- Human feedback: The user study interface asked people to choose the more reasonable saliency explanation for a given bounding box between randomly labeled stronger and weaker models.The compared models were labeled Robot 1 or Robot 2 randomly for each pair.
- Human feedback: 50.2% vs 27.4% found YOLOv3 explanations better or more trustworthy than YOLOv3-Tiny explanations.
- Evaluation metrics: The appendix adapts deletion and insertion metrics to measure the drop in similarity between an explained detection and the model output under partial image occlusion.For classification, deletion removes pixels in decreasing saliency order and evaluates how quickly class probability drops.
- Bias analysis: Moving markers used to train a biased YOLOv3 detector can alter predictions, including missed detections, false positives, or changed bounding-box dimensions.D-RISE explanations show an artifact at the marker’s training location in the average saliency map.
- Class-level saliency maps: Average saliency maps were computed for all MS-COCO classes using both YOLOv3 and Faster-RCNN.The maps used 5000 masks for YOLOv3 on 40k validation images and 2000 masks for Faster R-CNN on 5k images because of higher computational costs.