Source-linked AI summary
TIDE: A General Toolbox for Identifying Object Detection Errors
Daniel Bolya, Sean Foley, James Hays, Judy Hoffman
TL;DR
Object-detection and instance-segmentation performance is usually summarized by mAP, but mAP entangles error types and can obscure their relative effects. TIDE introduces a six-error framework with individually applied corrective oracles, showing that isolated weighting avoids the misleading conclusions of progressive analysis. The resulting toolbox supports compact, interpretable diagnosis across models, datasets, and tasks.
Problem
mAP intertwines detection errors, making their individual effects difficult to diagnose and potentially obscuring application-specific priorities.
Method
TIDE bins false positives and false negatives into six types and weights each type by its individual mAP effect from the vanilla model state using corrective oracles.
Results
TIDE provides a compact, meaningful, and expressive error analysis applicable across models, datasets, and tasks, while avoiding the confounding effects of progressive computation.
Takeaways & Limitations
TIDE supports model comparison, quantitative justification of design choices, and finer analysis of model strengths and weaknesses.
Takeaways & Limitations
The paper leaves determining the upper bound for mAP on these datasets, including the effect of mislabelled examples, to future work.
Abstract
from arXiv · showhide
We introduce TIDE, a framework and associated toolbox for analyzing the sources of error in object detection and instance segmentation algorithms. Importantly, our framework is applicable across datasets and can be applied directly to output prediction files without required knowledge of the underlying prediction system. Thus, our framework can be used as a drop-in replacement for the standard mAP computation while providing a comprehensive analysis of each model's strengths and weaknesses. We segment errors into six types and, crucially, are the first to introduce a technique for measuring the contribution of each error in a way that isolates its effect on overall performance. We show that such a representation is critical for drawing accurate, comprehensive conclusions through in-depth analysis across 4 datasets and 7 recognition models. Available at https://dbolya.github.io/tide/
1 Introduction
TIDE addresses the difficulty of diagnosing object-detection and instance-segmentation errors from mAP by isolating six error types and their effects on overall performance. The toolkit is designed for compact, interpretable analysis across models, datasets, and tasks.
- Motivation: mAP intertwines error types, making it difficult to determine how much each error affects performance.The metric averages precision-recall behavior across classes and, for COCO, IoU thresholds from 0.50 to 0.95.
- Motivation: Error importance can vary by application, with classification especially relevant to tumor detection and localization especially relevant to robotic grasping.Understanding these relationships supports model design and model selection for a given task.
- TIDE: TIDE aims to summarize error types compactly, isolate each contribution, avoid dataset-specific annotations, include all predictions, and support finer analysis.These goals target comparisons across datasets and more detailed diagnosis when needed.
- Prior work: Existing toolkits lack all of these desired properties, with prior approaches limited by false-positive-only summaries, difficult plots, progressive computation, or manual annotations.The COCO toolkit in particular can misrepresent error contributions through progressive analysis.
- Contributions: Each TIDE error type becomes a meaningful number weighted by its effect on overall performance without mAP’s confounding factors.The resulting error set is applicable across models, datasets, and tasks.
- Evaluation and use: The authors apply TIDE across object detectors and instance segmenters, use it in ablation studies, and demonstrate fine-grained analysis of model strengths and weaknesses.The toolkit code is released for future use and development.
2 The Tools
TIDE decomposes detection failures into six error types and estimates each type’s individual effect on mAP by applying corresponding corrective oracles from the original model state. This avoids the misleading weighting produced by progressive error analysis.
- Computing mAP: mAP can arise from false positives, false negatives, or incorrect confidence calibration, while individual detections may combine classification and localization errors.These possibilities make diagnosis from mAP alone difficult.
- Weighting the Errors: The framework weights each error individually from the vanilla model performance while ensuring that fixing all errors yields 100 mAP.This retains confidence and false-negative information while keeping error magnitudes comparable.
- Main Error Types: TIDE bins false positives and false negatives into six types using IoU thresholds and allows classification or localization errors to pair with false negatives.The thresholds are tf=0.5 and tb=0.1 unless otherwise noted.
- Weighting the Errors: Each error type is associated with an oracle that corrects the relevant detection, suppresses duplicates, or removes an unassignable false positive.The oracle definitions operationalize how each error is fixed in the evaluation pipeline.
- Limitations of Computing Errors Progressively: Progressive error computation misrepresents error weights because later corrections start with fewer false positives and potentially higher recall.These changes amplify the measured precision and mAP effects of errors processed later.
- Limitations of Computing Errors Progressively: Swapping the order of background and classification corrections substantially changes the perceived background contribution despite unchanged error quantities.The authors report that computing background first greatly decreases its apparent contribution, making factual conclusions difficult.
- Limitations of Computing Errors Progressively: Progressive analysis is also conceptually unsuitable for independent error improvement because a model does not pass through a state where one error category is fully corrected before another.Residual errors generally remain across categories as a method improves.
3 Analysis
TIDE is applied across models, datasets, object scales, ablations, and annotation quality to expose error distributions and guide finer diagnosis. The analyses connect architectural choices and dataset properties to distinct detection and segmentation error patterns.
- Analysis scope: TIDE analyzes diverse detectors and segmenters across COCO, Pascal VOC, Cityscapes, and LVIS, including cross-model and cross-dataset comparisons.The study also examines object size and provides more granular analysis for complicated error cases.
- Validating design choices: HTC has the lowest classification and localization errors, the lowest missed-ground-truth error, and the highest background error among the considered models, excluding YOLACT++.These patterns align with HTC’s iterative refinement and foreground-versus-background detection module.
- Validating design choices: RetinaNet’s focal loss is associated with low background error while retaining slightly less missed-ground-truth error than Mask R-CNN.The design down-weights confident examples so the model can train on all background anchor boxes.
- Validating design choices: Error ablations reveal both intended improvements and side effects that a change in mAP alone cannot expose.Mask rescoring decreases localization error while harming calibration of other error types, including classification calibration.
- Comparing performance across datasets: Across datasets, background error decreases as annotation density increases, while densely annotated datasets show different dominant challenges.Cityscapes is strongly affected by missed small objects, whereas LVIS is affected by classification-related missed ground truths; COCO and LVIS also reverse the relative difficulty of mask and box localization.
- Unavoidable errors: Among the top 100 confident Mask R-CNN errors on COCO, bad annotations account for 30 localization errors and 50 background errors.Examples include oversized boxes, omitted crowd annotations, and unlabeled objects.
4 Conclusion
The work defines error types and ties them to overall performance while minimizing confounding variables. It applies this framework to evaluate designs, compare attributes, and reveal dataset properties.
- TIDE defines meaningful error types and connects them to overall performance while minimizing confounding variables.
- The framework evaluates design decisions, compares performance on object attributes, and reveals properties of datasets.
- The authors aim for TIDE to isolate detection errors, improve interpretability of design decisions, and clarify model strengths and weaknesses.
A.1 Oddities of ∆AP
The appendix describes unintuitive properties of error-contribution AP changes. Individual contributions remain useful for comparison, but they do not generally add up to total performance changes because AP is non-additive across jointly fixed errors.
- Non-additivity of ∆AP: Error-type improvements generally do not sum to the observed overall AP improvement.For MS R-CNN, AP50 improves by +0.2 while the summed main-error changes predict +0.1; special errors predict −0.3.
- Interpretation: Despite these oddities, relative error magnitudes can still be compared, provided analysts keep the non-additivity in mind.
- Non-additivity of ∆AP: Summing individual ∆AP values does not generally equal 100 − AP.Mask R-CNN and YOLACT++ examples produce totals of 91.8 and 90.2 rather than 100.
- Progressive correction: The mismatch arises because errors are not computed progressively, and fixing multiple errors together can yield a larger mAP gain than fixing them separately.
- Interaction effects: In most cases, jointly measured error contribution exceeds the sum of separate contributions because conditional and unconditional AP changes differ.
A.2 AP0.5:0.95
The appendix examines error types across IoU thresholds for Mask R-CNN on COCO. Higher thresholds emphasize localization and false negatives, shaping the behavior of AP 0.5:0.95 and favoring methods with stronger localization calibration.
- Threshold effects: Localization error responds most strongly as the foreground IoU threshold increases, while false negatives become more important than false positives.The evaluation uses thresholds 0.5, 0.6, 0.7, 0.8, and 0.9 with Mask R-CNN detections on COCO.
- Metric implications: COCO and CityScapes averaging is heavily biased toward localization errors and, to a lesser extent, false negatives.This helps explain why localization-calibrated mask rescoring gains substantially on AP 0.5:0.95 despite little change at AP50.
B Implementation Details
The appendix records design and implementation details that the main paper could not fully explain because of space constraints.
- The appendix discusses design choices omitted from the main paper because of space constraints.
- It also covers implementation details that were not fully explained in the main paper.
- The section serves as supplementary explanation for material left out of the main presentation.
B.1 Defining the Missed GT Oracle
The paper treats the definition of a fixed missed-ground-truth oracle as difficult because adding a true positive requires assigning it a score. Evaluated scoring choices produce skewed or detector-dependent error contributions, motivating removal of ground truths instead.
- Oracle definition: Choosing an oracle for missed ground truth is difficult because adding a true positive requires selecting its score.The paper instead removes true negatives, thereby lowering the number of ground truths.
- Fixed scores: A score of 1 produces very high missed-GT values and likely overweights their contribution by assuming perfect calibration.This places the added detections at the beginning of the sorted list.
- Fixed scores: A score of −∞ makes missed-GT values depend heavily on the number of low-scoring detections produced by the model.Detectors that produce many low-scoring detections can therefore receive disproportionately small missed-GT values.
- Score-based alternatives: Sampling the mean or a uniformly sampled existing score produces the same skewed results as assigning −∞.These alternatives remain sensitive to the detector’s score distribution.
- Conclusion: Detector score distributions cannot be trusted as accurate because some models flood predictions with low-scoring detections.Consequently, the appropriate score for a newly added missed-GT prediction cannot be determined reliably.
B.2 Breaking Ties in Error Assignment
TIDE resolves ambiguous error assignments with an explicit priority rule. Such ties arise when one prediction has localization and classification errors relative to different ground truths, and the implementation assigns localization first.
- Tie cases: A prediction can have localization and classification errors relative to different ground truths because the two IoUmax calculations use different classes of ground truth.Classification uses a different-class ground truth, while localization uses a same-class ground truth.
- Tie cases: Approximately 0.78% of Mask R-CNN predictions on COCO produce these ties, so TIDE handles them with a defined rule.The reported frequency is low, but the assignment still needs to be specified.
- Tie-breaking rule: TIDE prioritizes localization over classification in the illustrated tie, choosing to trust classification more than localization.The authors describe this choice as largely arbitrary while noting that a decision is necessary.
- Implementation: Error types are assigned before ΔAP is computed, so tie breaking is separate from progressive error computation.All positives and negatives are accounted for before the performance contribution is calculated.
C COCO Instance Segmentation Summary
The appendix provides summary plots for COCO instance segmentation and reproduces the corresponding detection results from the main paper.
- COCO summary: Figure 11 summarizes errors for COCO instance segmentation while also reproducing the detection results from the main paper.The figure extends the main-paper summary to include instance segmentation.
D More Misannotated COCO Ground Truth
The COCO analysis finds that many highly confident errors involve misannotated or ambiguous ground truth. These annotation problems can affect both evaluation and training, making dataset quality a consequential part of model analysis.
- Sampling: The study examines the top 100 most confident errors in each error type, using a uniform random sample for missed ground truths.The sampled examples include misannotated and ambiguously annotated cases.
- Annotation categories: Misannotations are categorized as missing crowd labels, bad annotations, or ambiguous annotations.Examples include incorrect classes, incorrectly drawn boxes, nonexistent ground truth, and questionable omissions.
- Findings: Two thirds of the 100 most confident “both” errors are misannotated ground truth, while localization and background errors also contain many misannotations.Forgotten crowd labels are a surprisingly common source of localization errors.
- Training-set impact: Misannotations also occur in the training set, so incorrectly unlabeled crowd boxes can become training examples.The same annotation defects therefore affect more than evaluation alone.
- Interpretation: The analysis cautions that some errors may be caused by the dataset rather than the model and urges researchers not to treat datasets as black-box benchmarks.The authors state that the dataset can matter as much as the method.
- Toolkit update: In toolkit version 1.0.1, detections ignored in mAP calculation are included in error calculation, substantially changing LVIS results but barely affecting other datasets.On LVIS, the change replaces almost all missed error with classification error.
E Using Ignored Detections for Error Computation
The paper identifies an oversight in LVIS error computation involving ignored detections, then reports a toolkit fix that substantially changes the error interpretation. Including ignored detections has little effect on datasets such as COCO.
- LVIS error computation: An error-computation oversight caused the paper’s ECCV2020 conclusions on LVIS to be wrong.During LVIS mAP calculation, most detections corresponding to classes not guaranteed to appear in the image were ignored.
- Toolkit correction: In v1.0.0, affected misclassification errors were marked as missed detections because no non-ignored detection could match the ground truth.Toolkit version v1.0.1 fixed this matching issue.
- Toolkit correction: v1.0.1 changed most LVIS missed-ground-truth errors to classification errors, producing a more optimistic view of detectors on LVIS.The resulting change is shown in Fig. 12.
- Other datasets: On COCO, including ignored detections changed ΔAP by less than 0.4 for every error type.Other datasets likewise show relatively little change because they use ignored regions less heavily.