Source-linked AI summary
UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detection
Bumsoo Kim, Taeho Choi, Jaewoo Kang, Hyunwoo J. Kim
TL;DR
HOI detection must capture interactions beyond individual object detection, but prior sequential object-pairing pipelines create an inference bottleneck. UnionDet uses a one-stage union-level detector to directly capture interaction regions, reporting 4×–14× faster interaction prediction and state-of-the-art performance on V-COCO and HICO-DET. Its union-level detection design addresses challenges including human-region bias and inaccurate standard IoU matching.
Problem
Prior HOI detectors sequentially detect individual objects and then pair them to predict interactions, creating an inference-time bottleneck.
Method
UnionDet is a one-stage meta-architecture that directly detects human-object interaction union regions and pairs instead of performing separate post-detection pairwise inference.
Results
4×–14× speed-up in interaction prediction is reported while surpassing state-of-the-art performance on V-COCO and HICO-DET.
Takeaways & Limitations
The architecture supports near-real-time, unified HOI detection, is compatible with existing one-stage object detectors, and is end-to-end trainable.
Takeaways & Limitations
Vanilla union-branch prediction remains biased toward human regions and can suffer noisy learning when multiple union regions overlap.
Abstract
from arXiv · showhide
Recent advances in deep neural networks have achieved significant progress in detecting individual objects from an image. However, object detection is not sufficient to fully understand a visual scene. Towards a deeper visual understanding, the interactions between objects, especially humans and objects are essential. Most prior works have obtained this information with a bottom-up approach, where the objects are first detected and the interactions are predicted sequentially by pairing the objects. This is a major bottleneck in HOI detection inference time. To tackle this problem, we propose UnionDet, a one-stage meta-architecture for HOI detection powered by a novel union-level detector that eliminates this additional inference stage by directly capturing the region of interaction. Our one-stage detector for human-object interaction shows a significant reduction in interaction prediction time 4x~14x while outperforming state-of-the-art methods on two public datasets: V-COCO and HICO-DET.
1 Introduction
UnionDet addresses the inference bottleneck in HOI detection by directly detecting interaction regions in a one-stage architecture. It reports 4×–14× faster interaction prediction while achieving state-of-the-art performance on two public datasets.
- HOI detection requires localizing and classifying interactions between humans and objects beyond detecting individual objects.
- Prior methods sequentially detect individual objects and associate them into pairs, creating an inefficient multi-stage pipeline and an inference-time bottleneck.This sequential design also makes end-to-end training impossible.
- UnionDet is a one-stage meta-architecture with a union-level detector that directly captures the interaction region and interacting human-object pairs.It replaces separate post-detection pairwise neural-network inference with direct union-level detection.
- The method studies union-level detection challenges including human-region bias, inaccurate standard IoU matching, and regions containing multiple interactions or more than two objects.
- UnionDet introduces training techniques for union-level detection, including union anchor labeling, target object classification loss, and union foreground focal loss.
- 4×–14× speed-up in interaction prediction accompanies state-of-the-art performance on V-COCO and HICO-DET.
2 Related Work
Related work established faster one-stage object detection and increasingly rich HOI representations, but individual-box cues can lack contextual information.
- 2.1 One-Stage Object Detection: One-stage object detectors were proposed to accelerate detection, although faster detection can involve considerable accuracy loss and foreground-background class imbalance.Focal loss was introduced to reduce the effect of easy negatives in densely sampled anchor boxes.
- 2.2 Human-Object Interaction Detection: HOI detectors have incorporated human body parts, human appearance, instance appearance, and spatial relationships between human-object pairs.InteractNet used an action-specific density map to localize target objects from a detected human, while iCAN addressed missing contextual information from individual boxes.
3 Method
UnionDet is a one-stage HOI detector that directly detects human-object interaction regions while retaining instance-level detection for benchmark-required localization. Its Union Branch addresses human-region bias, inaccurate IoU matching, and overlapping unions through specialized labeling and losses, adding only marginal inference time.
- Architecture: UnionDet combines a Union Branch that directly detects interaction regions with a parallel Instance Branch for object and action detection.The two branches share backbone features and are trained jointly.
- Challenges: Union-level detection must handle human-region bias, inaccurate standard IoU matching, and union regions containing multiple interactions or objects.These challenges arise because every HOI union contains a human, remote objects may be missed despite high IoU, and unions can overlap.
- Union Branch: The Union Branch uses union anchor labeling, target object classification loss, and union foreground focal loss to address these challenges.Anchor labeling requires suitable inclusion of the union, human, and object regions; target classification encourages coverage of the target object, while the modified focal loss handles overlapping unions.
- Union Branch: The union-instance matching score replaces standard IoU to evaluate whether predicted regions cover the interacting boxes appropriately.The score uses box-area relationships and the tightest box covering the relevant region.
- Inference: ∼9ms is the marginal end-to-end inference-time increase over RetinaNet with ResNet50-FPN, because final triplet scoring requires no additional neural-network inference.Although the score calculation has O(n^3) complexity in the number of detections, its measured calculation time is less than 1ms.
4 Experiments
UnionDet is evaluated on V-COCO and HICO-DET through qualitative, quantitative, ablation, and inference-time analyses. The experiments show that its union-level detector handles ambiguous and spatially varied interactions while achieving fast, accurate HOI detection.
- Union-level detection: Union-level detections disambiguate confusing human-object pairs sharing the same action and target-object types.The union region uses interaction context to group correct pairs when instance-level appearances are insufficient.
- Interactions in various distances: UnionDet detects interactions involving included, adjacent, distant, and remote target objects, including small remote objects such as tennis balls and frisbees.Target Object Classification addresses the bias toward human regions in union-level detection.
- HOI detection results: The final model combines both branches to detect one-to-one, many-to-one, one-to-many, and many-to-many interaction relationships.The qualitative results visualize instance-level human and object detections together with predicted union regions.
- Performance analysis: UnionDet surpasses state-of-the-art performance on V-COCO and achieves state-of-the-art performance under both official HICO-DET evaluation settings.The V-COCO comparison covers both Scenario#1 and Scenario#2, while HICO-DET reports Default and Known Object settings.
- Ablation study: Ablation results show that the Union Branch and the proposed training components improve overall HOI detection performance.The evaluated components include foreground focal loss, target object classification loss, and the union matching function.
- Interaction prediction time: 4× ∼14× speed-up in additional interaction-prediction inference time is achieved relative to baseline HOI models requiring 40ms ∼130ms after object detection.UnionDet adds 9.06 ms (11.7%) over a base object detector and reaches 77.6 ms end-to-end inference time on a single Nvidia GTX1080Ti GPU.
5 Conclusions
UnionDet performs object detection and human-object interaction prediction in parallel within a one-stage, end-to-end trainable architecture. It maintains comparable state-of-the-art performance while achieving near-real-time inference and broad compatibility with one-stage object detectors.
- UnionDet performs action classification and union region detection in parallel with object detection.
- The architecture is end-to-end trainable and generally compatible with existing one-stage object detectors.
- UnionDet achieves the fastest inference time while maintaining comparable performance with state-of-the-art methods.
- The unified detector operates at near-real-time frame rates without switching models or saving and loading intermediate results across stages.