Source-linked AI summary
GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection
Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, Si Liu
TL;DR
HOI detection must solve human-object association and interaction understanding, but existing query-driven methods impose matching or representation limitations and struggle with long-tailed and zero-shot cases. GEN-VLKT uses guided embeddings in a two-branch detector and transfers CLIP knowledge through classifier initialization and visual mimicry. It reports leading performance on regular and zero-shot benchmarks, including a 5.05 mAP gain on HICO-Det over QPIC.
Problem
HOI detectors must associate interactive humans and objects while understanding their interactions, yet existing approaches face complex post-matching, task-specific feature differences, long-tailed distributions, and zero-shot discovery challenges.
Method
GEN-VLKT combines a two-branch Guided Embedding Network without post-matching with VLKT, which transfers CLIP knowledge through text-initialized classifiers and visual embedding mimicry.
Results
GEN-VLKT achieves leading regular and zero-shot performance on HICO-Det and V-COCO, including a 5.05 mAP gain on HICO-Det over QPIC.
Takeaways & Limitations
Guided embeddings simplify association, while CLIP transfer supports interaction understanding and extension to novel HOI categories without extra inference cost.
Takeaways & Limitations
The authors note that global visual-feature mimicry from CLIP may be insufficient for dense HOI understanding and plan to improve the mimic strategy.
Abstract
from arXiv · showhide
The task of Human-Object Interaction~(HOI) detection could be divided into two core problems, i.e., human-object association and interaction understanding. In this paper, we reveal and address the disadvantages of the conventional query-driven HOI detectors from the two aspects. For the association, previous two-branch methods suffer from complex and costly post-matching, while single-branch methods ignore the features distinction in different tasks. We propose Guided-Embedding Network~(GEN) to attain a two-branch pipeline without post-matching. In GEN, we design an instance decoder to detect humans and objects with two independent query sets and a position Guided Embedding~(p-GE) to mark the human and object in the same position as a pair. Besides, we design an interaction decoder to classify interactions, where the interaction queries are made of instance Guided Embeddings (i-GE) generated from the outputs of each instance decoder layer. For the interaction understanding, previous methods suffer from long-tailed distribution and zero-shot discovery. This paper proposes a Visual-Linguistic Knowledge Transfer (VLKT) training strategy to enhance interaction understanding by transferring knowledge from a visual-linguistic pre-trained model CLIP. In specific, we extract text embeddings for all labels with CLIP to initialize the classifier and adopt a mimic loss to minimize the visual feature distance between GEN and CLIP. As a result, GEN-VLKT outperforms the state of the art by large margins on multiple datasets, e.g., +5.05 mAP on HICO-Det. The source codes are available at https://github.com/YueLiao/gen-vlkt.
1. Introduction
HOI detection requires both associating interactive humans and objects and understanding their interactions. GEN-VLKT addresses association with guided embeddings and interaction understanding with CLIP-based visual-linguistic knowledge transfer.
- HOI detection represents activities as <Human, Object, Verb> triplets, requiring human-object localization and interaction recognition.
- Query-based detectors face costly post-matching in two-branch pipelines and task-specific feature differences in single-branch pipelines.
- GEN preserves two task-specific decoder branches while using guided embeddings to associate human-object pairs during the forward pass.Independent human and object queries use position Guided Embeddings, while instance Guided Embeddings guide interaction queries.
- VLKT transfers CLIP knowledge to interaction understanding through text-driven classifier initialization and visual knowledge transfer.HOI and object labels are converted into text descriptions and encoded with CLIP for classifier initialization.
- 5.05 mAP on HICO-Det and 5.28 AP on V-COCO exceed the previous state-of-the-art method QPIC.GEN-VLKT also reports a 108.12% relative mAP gain for unseen-object zero-shot detection over ATL.
2. Related Works
Prior HOI research includes bottom-up and top-down association methods, zero-shot approaches, and visual-linguistic knowledge transfer. GEN differs from related two-decoder designs through parallel guided association and separate human and object queries.
- HOI detection: Bottom-up HOI pipelines detect humans and objects before associating pairs and inferring interaction types with an additional classifier.
- HOI detection: GEN and CDN both use two decoders, but GEN runs its instance and interaction decoders together rather than serially.
- HOI detection: GEN uses isolated human and object queries with positional embeddings, whereas CDN entangles them in one instance query.
- Zero-shot HOI detection: Zero-shot HOI methods generate unseen triplets by factorizing features, composing novel samples, fabricating objects, or modeling object affordances.
- Visual-linguistic knowledge: Visual-linguistic models provide a basis for HOI recognition, including CLIP-based classifier initialization explored by DEFR.
3. Methods
GEN uses a two-branch decoder with guided embeddings to associate human-object pairs before interaction classification, while VLKT transfers CLIP knowledge into interaction understanding.
- Guided Embedding Network: GEN combines a visual encoder with instance and interaction decoders in a one-stage two-branch architecture.The instance decoder detects humans and objects, while the interaction decoder predicts HOI categories.
- Guided Embedding Network: Two independent human and object query sets receive shared positional embeddings so queries at the same position represent a pair.The concatenated instance queries are used to predict human-object bounding-box pairs.
- Guided Embedding Network: Interaction queries are dynamically generated from corresponding instance-decoder outputs, guiding the interaction decoder toward the associated human-object pairs.Each interaction-decoder layer consumes interaction queries and returns decoded interaction features and HOI categories.
- Visual-Linguistic Knowledge Transfer: VLKT converts HOI and object labels into CLIP text embeddings and uses them to initialize interaction and object classifiers.The embeddings are extracted offline from phrase descriptions, providing classifier weights that are fine-tuned for the dataset.
- Visual-Linguistic Knowledge Transfer: VLKT aligns GEN interaction features with CLIP visual embeddings through visual embedding mimicry during training.CLIP acts as the teacher, GEN as the student, and the global student embedding is formed by average pooling interaction-query features.
- Training and Inference: The final training loss combines the matching cost with a weighted mimic loss, while visual embedding mimicry is removed during inference.Inference combines instance object scores and interaction scores before triplet NMS produces the final predictions.
4. Experiments
GEN-VLKT is evaluated on HICO-Det and V-COCO, including regular and zero-shot HOI detection settings. It reports strong benchmark performance and improvements across unseen composition, object, and verb categories, while ablations examine VLKT components.
- Experimental Settings: GEN-VLKT is evaluated on HICO-Det and V-COCO using mAP-based HOI metrics and zero-shot category splits.HICO-Det includes Full, Rare, and Non-Rare categories; zero-shot experiments evaluate Unseen Composition, Unseen Object, and Unseen Verb settings.
- Regular HOI Detection: GEN-VLKTs achieves a 16.10% relative mAP gain over QPIC on HICO-Det, with a 4.68 mAP margin.GEN-VLKTs reaches 29.25 mAP on Rare categories, exceeding AS-Net by 5.00 mAP; GEN-VLKTl reaches 34.95 mAP.
- Regular HOI Detection: GEN-VLKTs achieves role AP 62.41 on V-COCO Scenario 1 and 64.46 on Scenario 2, exceeding QPIC-R50 by 3.61 and 3.46 mAP, respectively.The reported promotion is smaller than on HICO-Det because V-COCO has insufficient training samples for its 263-category classification.
- Zero-Shot HOI Detection: 38.85% and 22.41% relative mAP gains are reported over FCL for full categories in rare-first and non-rare-first Unseen Composition settings.In rare-first UC, mAP increases from 13.16 to 21.36 compared with FCL, and exceeds the baseline by 8.84 mAP.
- Zero-Shot HOI Detection: 95.95% and 108.12% relative mAP gains over ATL are reported for full and unseen categories in Unseen Object evaluation.The comparison with the baseline is also reported to show significant improvement for unseen categories.
- Zero-Shot HOI Detection: 55.03% relative promotion is reported for unseen categories in the Unseen Verb setting compared with the baseline.The setting targets discovery of novel action categories.
- Ablation Study: Ablations show that text embedding initialization improves Rare HOIs, object-classifier transfer adds 0.38 mAP, and adding mimic loss further boosts performance.Among tested mimic losses, L1 achieves a 1.66 mAP gain and outperforms L2 or their summed use.
5. Conclusion
GEN-VLKT improves query-based HOI detection through separate designs for association and interaction understanding. Its evaluation includes ablations on HICO-Det, while the authors note that global CLIP feature mimicry may remain insufficient for dense understanding.
- GEN uses a two-branch framework that removes post-matching through a guided embedding mechanism for association.
- Ablation experiments evaluate network architecture, VLKT training strategies, and mimic-loss choices on HICO-Det’s ‘s’ model.
- VLKT adopts CLIP to enhance interaction understanding in GEN-VLKT, which achieves leading performance on regular and zero-shot HICO-Det and V-COCO settings.