Source-linked AI summary
Diagnosing Error in Temporal Action Detectors
Humam Alwassel, Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem
TL;DR
Temporal action localization has improved, but how close it is to reliably delimiting actions and what drives method differences remain open questions. This paper introduces diagnostic analyses for action detectors and applies them to leading ActivityNet challenge methods, finding that context, coverage, and length are more influential than annotator agreement or instance count.
Problem
Despite progress in temporal action localization, it remains unclear how accurately methods delimit action boundaries and what makes methods or instances difficult.
Source: §1 · pp. 1–2Method
The authors add context and temporal-agreement annotations, analyze false-positive profiles, and assess detector performance across action characteristics and missed detections.
Source: §1 · p. 3, §3.1 · p. 6, §5 · p. 10Results
Across four methods, sensitivity trends were similar: context, coverage, and length mattered more than agreement or instance count, and three of four would benefit most from temporal-context reasoning.
Source: §6 · pp. 12–13Takeaways & Limitations
The diagnostic tool identifies method-specific false-positive weaknesses and supports analysis of missed detections, including scoring or classifier shortcomings.
Source: §5 · p. 11, §8 · p. 14
Abstract
from arXiv · showhide
Despite the recent progress in video understanding and the continuous rate of improvement in temporal action localization throughout the years, it is still unclear how far (or close?) we are to solving the problem. To this end, we introduce a new diagnostic tool to analyze the performance of temporal action detectors in videos and compare different methods beyond a single scalar metric. We exemplify the use of our tool by analyzing the performance of the top rewarded entries in the latest ActivityNet action localization challenge. Our analysis shows that the most impactful areas to work on are: strategies to better handle temporal context around the instances, improving the robustness w.r.t. the instance absolute and relative size, and strategies to reduce the localization errors. Moreover, our experimental analysis finds the lack of agreement among annotator is not a major roadblock to attain progress in the field. Our diagnostic tool is publicly available to keep fueling the minds of other researchers with additional insights about their algorithms.
1 Introduction
Despite improving temporal action localization, basic questions about boundary accuracy, method differences, and hard-to-localize actions remain open. The paper proposes quantitative diagnosis beyond a single metric and demonstrates it on four leading ActivityNet challenge detectors.
Contributions
- The tool analyzes false-positive sources and their impact, metric sensitivity to instance characteristics, and how those characteristics relate to miss rates.These analyses address what errors methods make, which instance traits affect performance, and what instances they miss.Source: Figure 1 · p. 2, §1 · p. 2
Relation to Existing Studies
- The paper presents the first characterization of localization errors for temporal action localization in videos, distinct from prior diagnosis of frame- or video-level action recognition.The cited recognition study examines attributes such as verbs, objects, poses, and interactions, mainly on short streams.Source: §1 · pp. 2–3
Contributions
- The authors collect context and temporal-agreement annotations, grouping instances by six characteristics: context size, context distance, agreement, coverage, length, and instance count.The resulting annotations extend ActivityNet for analysis of temporal context and boundary agreement.Source: §1 · p. 3
- The tool includes annotations, software, and techniques, and is demonstrated on the four top entries in the 2017 ActivityNet action-localization challenge.The analysis examines detectors’ performance using the proposed diagnostic approach.Source: §1 · p. 3
2 Preliminaries
The paper evaluates temporal action localization on ActivityNet v1.3 and uses normalized average precision to compare performance across subsets. It also describes the challenge methods selected to demonstrate the diagnostic tool.
Evaluation Framework
- ActivityNet v1.3 is the main test bed, with 20K diverse videos averaging four minutes; supplementary analysis also uses THUMOS14.The authors cite its held-out test and validation sets, open evaluation framework, and annual competition as reasons for choosing it.Source: §2 · p. 3
- Average-mAP averages per-class mAP values across tIoU thresholds from 0.5 to 0.95 in 0.05 steps; a detection is a true positive when its tIoU meets the threshold.AP is the interpolated area under the precision-recall curve, computed per category before averaging into mAP.Source: §2 · pp. 3–4
- Average-mAPN is slightly higher than average-mAP, while top-10G predictions yield performance similar to using all predictions.Table 1 reports both metrics for all predictions and for the top-10G predictions, where G is the number of ground-truth instances.Source: Table 1 · p. 4
- Normalized mAP uses precision adjusted by N, set to the average ground-truth segments per class, to compare subsets with differing class-instance counts.The paper reports average-mAPN as its localization metric.Source: §2 · p. 4
Algorithms
- All four challenge methods use proposals followed by classification; SC won by a 2% average-mAP margin, while CES was runner-up.SC emphasizes proposals; CES uses temporal grouping and an SSN classifier that balances segment and surrounding-context information.Source: §2 · pp. 4–5
- IC ranked third with sliding-window proposals and pose-influenced classification; BU was the most innovative entry, extending Faster RCNN to temporal localization.BU relies on RGB streams and 3D convolutions pretrained on Sports-1M.Source: §2 · p. 5
3 Dataset Characterization
The section augments ActivityNet v1.3 with temporal-context and boundary-agreement annotations, then defines six instance characteristics for analyzing action localization conditions.
3.1 Online User Studies
- In 30K tasks, Turkers guessed the action correctly in 90.8%, suggesting temporal reasoning is needed for localization in such scenarios.The study sampled six non-overlapping five-second clips around each action instance.Source: §3.1 · p. 6
- Across four annotations per instance, median pairwise tIoU agreement was 64.1%, with three new Turker labels collected for each ActivityNet action.The task was to annotate boundaries of an already spotted action.Source: §3.1 · p. 7
3.2 Definitions of Action Characteristics
- Only 6.9% of instances lacked temporal context, while 58.4% had more than three context glimpses; 69.9% had glimpses far away.Context size counts glimpses; context distance records the furthest distance at which a glimpse exists.Source: §3.2 · p. 7
- At least Mid agreement covered 83.8% of instances; Extra Small and Extra Large coverage were 42.4% and 27.4%, and 54.4% were Small by duration.Half the dataset contained a single same-class instance per video, and instance counts gradually decreased with instance duration.Source: §3.2 · p. 8
4 Categorization of Temporal Localization Errors
The paper defines five false-positive categories based on label correctness and temporal overlap, while treating missed ground-truth instances as false negatives. The taxonomy is intended to identify which errors matter for temporal localization.
4 Categorization of Temporal Localization Errors
- Five false-positive categories distinguish prediction errors, while missed ground-truth instances are analyzed separately as false negatives.Figure 4 illustrates the categories and a missed detection.Source: §4 · pp. 8–9, Figure 4 · p. 9
- Double Detection Error occurs when a threshold-matching, correctly labeled prediction duplicates a ground-truth instance already matched by a higher-scoring prediction.The authors link this error to ActivityNet’s evaluation, which penalizes double detections at high tIoU thresholds.Source: §4 · p. 9
- Wrong Label Error occurs when a prediction meets the tIoU threshold but assigns the ground-truth instance an incorrect label.The authors associate this error often with weakness in the action classification module.Source: §4 · p. 9
- Localization Error is a correctly labeled prediction with at least 0.1 tIoU that fails the required α tIoU threshold.The authors associate this error typically with the localization module and/or temporal feature representation.Source: §4 · p. 9
- Confusion Error combines an incorrect label with at least 0.1 tIoU, without meeting the required α tIoU threshold.The authors attribute it to the same sources as Wrong Label and Localization Errors.Source: §4 · p. 9
- Background Error is a prediction with less than 0.1 tIoU overlap with any ground-truth instance.The authors note that this error can arise in large percentages from a weakness in the prediction scoring scheme.Source: §4 · p. 9
5 False Positive Analysis
The analysis profiles false-positive errors among ranked predictions from four state-of-the-art methods and measures the impact of each error type. Localization errors have the largest impact on average-mAPN, while the profiles reveal distinct error patterns across methods.
5 False Positive Analysis
- The authors analyze four methods (SC, CES, IC, BU), splitting each method’s per-class top-10G predictions into ten equal groups to profile five false-positive types and model error rates by prediction score.They choose top-10G because it is sufficiently large to show error-type trends; beyond it, methods have normalized precision < 0.05PN, where analysis is uninsightful as prediction quality degrades and background errors dominate.Source: §5 · p. 10
- Removing Localization Error predictions produces the largest average-mAPN improvement among the five error types across the four methods.Figure 5’s bottom panel presents the impact comparison.Source: §5 · p. 11
- SC has the lowest background error rate but a relatively high double-detection rate, whereas IC’s predictions beyond the top-G are mostly background errors.The authors attribute IC’s pattern to its scoring scheme and SC’s double detections to its proposal method and independent video-level classifier.Source: §5 · p. 11
6 Average-mAPN Sensitivity
The sensitivity analysis finds that temporal context, coverage, and instance length affect detector performance more than agreement or instance count. It also shows that three of four studied models would benefit most from stronger temporal-context reasoning.
6 Average-mAPN Sensitivity
- All four methods show similar performance-variation trends across action characteristics, unlike the methods’ differing false-positive profiles.The analysis filters out instances that do not exhibit each characteristic before measuring performance.Source: §6 · pp. 11–12
- Instances with context size 5–6 are associated with lower performance, which the authors attribute to similar surrounding actions confusing boundary placement.The passage presents this explanation as an argument rather than conclusive evidence.Source: §6 · p. 12
- Short instances, whether short in absolute duration or relative to video length, tend to be harder to detect, possibly because algorithms accumulate temporal evidence coarsely.Source: §6 · p. 12
- Higher human agreement on action boundaries is associated with performance gains, while lower agreement is associated with drops, but the authors find no conclusive explanation for this correlation.They suggest instance nature or dataset bias as possibilities.Source: §6 · p. 12
- Three of four studied models would benefit most from improved temporal-context reasoning, and all methods are more sensitive to context, coverage, and length than to agreement or instance count.The sensitivity profile compares the range of average-mAPN across each characteristic with its potential impact relative to overall performance.Source: §6 · pp. 12–13, Figure 6 · p. 12
7 False Negative Analysis
False-negative rates reveal difficulty across individual and paired action characteristics, including a pronounced challenge when videos contain multiple instances. Weak agreement, low coverage, short length, and large temporal context also align with higher miss rates.
7 False Negative Analysis
- Missed detections increase more than twofold when another instance is present in a video, identifying multiple instances as a substantial challenge.Source: §7 · p. 13
- Higher false-negative rates align with the sensitivity analysis for weak agreement, low coverage, short instances, and large temporal context; spreading context also increases boundary confusion.Source: §7 · p. 13
- Low coverage paired with large context, extremely weak agreement paired with large context, and low coverage paired with extremely weak agreement are especially difficult to detect.These combinations are difficult even when some are well represented in the dataset.Source: §7 · p. 14
8 Discussion and Conclusion
The paper uses a diagnostic tool to analyze four ActivityNet challenge approaches and characterize their failure modes. The analysis finds localization errors hurt all detectors, temporal context is a major sensitivity, and annotator agreement is not a major barrier to progress.
8 Discussion and Conclusion
- The diagnostic tool is applied to four approaches from the latest ActivityNet action localization challenge to examine false-positive sources and missed detections.Source: §8 · p. 14
- False-positive profiles vary across methods, with some showing scoring-function shortcomings and others classifier weaknesses.Source: §8 · p. 14
- Localization errors strongly hurt all detectors, while methods are very sensitive to temporal context.Source: §8 · p. 14
- The authors introduce six ActivityNet action characteristics—context size, context distance, agreement, coverage, length, and number of instances—and find annotator agreement is not a significant barrier to improving localization.Source: §8 · p. 14
- The authors plan to explore localization metrics that incorporate ambiguity in temporal action boundaries.Source: §8 · p. 14