Affordance Transfer Learning for Human-Object Interaction Detection

1. Introduction

• Rare or unseen interactions with novel objects, such as “ride tiger,” remain a challenge for HOI detection.Source: §1 · p. 1
• ATL composes affordance representations from existing HOIs with novel-object representations, transferring shared affordances to detect otherwise unseen interactions.For example, a shared “rideable” affordance between horse and tiger supports composing “ride tiger.”Source: §1 · p. 2
• The method also uses the HOI model to infer novel-object affordances by combining object representations with a feature bank of decoupled affordances.The combined representations are classified as HOIs, whose predictions are used to infer object affordances.Source: §1 · p. 2
• The authors report that the method improves recent state-of-the-art HOI detection methods and facilitates object-affordance recognition.Source: §1 · p. 2

2.1. HOI Understanding

• ATL composes affordance and novel-object representations, addressing VCL’s inability to compose human–novel-object interactions and its omission of affordance recognition with the HOI model.Source: §2.1 · p. 2
• HOI detection methods are categorized as two-stage or one-stage, with two-stage work including common, few-shot, and zero-shot approaches.Source: §2.1 · p. 2

2.2. Object Affordance

• Earlier affordance work includes learning from demonstrations, weakly supervised discovery from HOI data, functional-surface estimation, and action labeling from online video.Source: §2.2 · p. 2
• Figure 2 depicts ATL extracting affordance and object features, composing new HOIs with objects from additional datasets, and classifying real and composite HOIs with a shared classifier.Source: Figure 2 · p. 3

3.1. Overview

• Real and composite HOIs share an HOI classifier, while composite HOIs use object features extracted from additional object images alongside affordance features.A spatial HOI branch adds spatial-pattern representations to the two-stage framework.Source: §3.1 · p. 3

3.2. Affordance Transfer Learning

• Affordance transfer composes novel HOIs from additional-object features and decoupled affordances, assigning labels by intersecting object- and verb-compatible HOI labels and removing combinations outside the ground-truth label space.The method removes invalid combinations such as “ride dog” because their one-hot labels are all zeros.Source: §3.2 · pp. 3–4

3.3. Object Affordance Recognition

• For affordance recognition, object features are combined with a bank of affordance features, and classifier predictions are converted to affordances after filtering for labels represented in the bank.The bank is constructed from HOI datasets and uses at most 100 instances per affordance in HICO-DET in the reported experiment.Source: §3.3 · p. 4, Figure 3 · p. 4

3.4. Optimization and Inference

• Training jointly optimizes affordance-transfer, spatial-pattern, and HOI-classification losses end-to-end; at test time, the transfer module is unnecessary and final HOIs use spatial and verb-object predictions.The losses are binary cross-entropy losses, with hyperparameters balancing their contributions.Source: §3.4 · pp. 4–5

4.1. Datasets and Evaluation Metrics

• HOI-COCO covers 222 verb-object categories over 21 verbs and 80 objects; Object365 evaluation includes 12 manually labeled novel object types.HOI detection is evaluated with mean average precision; affordance recognition uses Precision, Recall, and F1-Score.Source: §4.1 · p. 5

4.2. Implementation Details

• Training uses a fine-tuned HICO-DET detector or a COCO-pretrained HOI-COCO detector, with random box augmentation and a cap on composite-HOI sampling.The model is trained for 1.2M iterations on HICO-DET and 300K on HOI-COCO.Source: §4.2 · pp. 5–6

4.3. HOI Detection

• 28.53% is reached with better object-detection results, while ATL outperforms recent state-of-the-art methods across HICO-DET categories.The study evaluates Full, Rare, and NonRare categories in Default and Known modes.Source: Table 1 · p. 6, §4.3 · p. 6
• 25.29% is achieved on HOI-COCO using both HICO-DET and COCO object images; COCO alone improves Rare-category performance by over 1.5% versus VCL and 2.9% versus baseline.Using COCO object images also improves NonRare performance more than the baseline relative to VCL.Source: Table 2 · p. 6, §4.3 · p. 6

4.4. Zero-Shot HOI detection

• 5.05% unseen-category performance is achieved by ATL (COCO) in novel-object detection, while box-only baseline and ATL (HICO-DET) results fall to 0; rare-first unseen results can also worsen.The authors attribute the rare-first limitation to fewer composite samples for tail affordances than for head HOIs.Source: Table 3 · p. 6, §4.4 · pp. 6–7

4.5. Object Affordance Recognition

• ATL improves affordance-recognition F1 by over 40% across datasets with COCO categories and over 30% on HOI-COCO novel classes; HICO-DET gains are nearly 10% overall and around 5% on novel classes.On HICO-DET, ATL also performs better than VCL for affordance recognition, although their HOI-detection performance is similar; Precision, Recall, and F1-Score are threshold-sensitive.Source: Table 4 · p. 7, §4.5 · p. 7

4.6. Ablation Studies

• Two object images per batch perform best; COCO object images help HOI-COCO under domain shift, while ATL is sensitive to poor boxes on HICO-DET.On HICO-DET, ATL also outperforms object-detection-derived affordances across confidence levels, especially at lower detected-box confidence.Source: §4.6 · pp. 7–8, Figure 4 · p. 8

4.7. Qualitative Results

• In qualitative novel-object examples, the baseline fails to recognize the object, while ATL detects the HOI involving that unseen object.Figure 5 compares the top-five results and highlights correct results in red.Source: Figure 5 · p. 8, §4.7 · p. 8

• ATL composes objects from object images with affordances from HOI images to transfer affordances to novel objects.Source: §5 · p. 8
• ATL facilitates long-tailed HOI detection, especially for interactions involving novel objects.Source: §5 · p. 8
• ATL significantly improves the HOI detection model's performance on object affordance recognition.Source: §5 · p. 8

A. Overview

• The overview presents ATL as an approach to human-object interaction understanding and object understanding in a unified way.It also points readers to supplementary sections on the method, comparisons, and additional results.Source: A. Overview · p. 10
• Figure 6 provides examples relating HOI and affordance.Source: Figure 6 · p. 10

B. More Examples of HOI and Object Affordance

• HOI annotations also show object affordances, and combining affordance representations with corresponding objects lets the HOI model recognize affordances of novel objects.Source: B. More Examples of HOI and Object… · p. 10

C. Affordance Transfer Learning for One-Stage HOI detection

• The implementation uses two object images and four HOI images per batch with two GPUs, and defines its baseline as the model without compositional learning loss.Source: C. Affordance Transfer Learning for… · pp. 10–11
• Table 8 studies object-affordance recognition across object-image datasets and batch image counts on HICO-DET.The caption identifies Val2017, a COCO-labeled Object365 subset, and novel non-COCO Object365 classes.Source: Table 8 · p. 11
• Table 9 compares affordance-recognition mAP with and without a union verb representation across object datasets and novel classes.Without union verb, the verb representation is extracted from the human box.Source: Table 9 · p. 11
• The one-stage method predicts 117 verbs and combines them with object-detection results to construct HOI predictions.It composes object and verb features across HOI and object images, while retaining object-detection optimization for object images.Source: C. Affordance Transfer Learning for… · pp. 10–11

D. Supplementary Description

• HICO-DET, Val2017, and COCO-labeled Object365 evaluation objects come from 12 novel classes, while the Novel Classes category uses non-COCO classes.The affordance annotations for the 12 new object types follow HICO-DET and HOI-COCO, respectively.Source: D. Supplementary Description · p. 11
• Nearly 900,000 COCO object instances versus around 100,000 HICO instances may explain why adding COCO object images to HOI images did not improve HICO-DET results.Source: D. Supplementary Description · p. 11
• For novel-object zero-shot evaluation, the authors remove some COCO object images to balance the many composite HOIs for seen HOIs.Source: D. Supplementary Description · p. 11
• The baseline fuses HOI-detection results into object-detection results, and the authors fine-tune ATL models for simplicity.ATL with HICO data as object images converges faster because it uses fewer training data than ATL with COCO.Source: D. Supplementary Description · p. 11

E. Additional Ablation Study

• More object images per batch improve affordance recognition on Non-COCO classes, while two images perform better than one or three among COCO categories.The authors interpret the multi-image result as better generalization to novel classes.Source: E. Additional Ablation Study · p. 11
• Using human-box rather than union-box verb features lowers ATL affordance recognition by over 5% on COCO, Object365, and HICO-DET, while baseline recognition improves.The two feature choices yield similar HOI-detection performance; the authors suggest union boxes include noise outside the human and object boxes.Source: E. Additional Ablation Study · p. 11
• Affordance-recognition results stabilize after 20 verb instances, so the feature bank need not store many affordance templates.For each affordance, the ablation samples M instances when more than M are available and uses all instances otherwise.Source: E. Additional Ablation Study · p. 12

F. Non-COCO classes

• ATL recognizes affordances for non-COCO objects without observed interactions by combining affordance representations with object features.The evaluation selects 12 non-COCO Object365 classes; Table 12 lists their affordances on HICO-DET.Source: F. Non-COCO classes · p. 12, Table 12 · p. 12

G. Additional Results and Comparision

• The compositional-learning approach consistently improves the baseline across categories in affordance-recognition mAP.The authors adopt mAP because Recall, Precision, and F1 were not robust, with F1 sensitive to confidence.Source: G. Additional Results and Comparision · p. 12
• Additional results for recent HOI-detection methods are provided in Table 16 because of space limitations in the main paper.The supplied passage does not state comparative outcomes for those methods.Source: G. Additional Results and Comparision · p. 12

H. Discussion with Prior Approaches

• ATL extends VCL by composing verbs and objects from object-detection datasets without HOI annotations, and its compositional HOI network can also recognize affordances.The discussion also reports improved affordance generalization on a new dataset with more data.Source: H. Discussion with Prior Approaches · p. 12
• FCL can outperform ATL on some zero-shot HOI settings by generating more balanced objects per verb, but its model cannot recognize affordances.The discussion frames ATL and FCL as complementary in these capabilities.Source: H. Discussion with Prior Approaches · pp. 12–13
• Table 16 provides additional illustrations of recent HOI-detection approaches.The discussion notes these results are omitted from the main paper because of space limitations.Source: Table 16 · p. 13
• Figure 7 shows examples of non-COCO classes and their affordances, including affordance labels such as carriable, holdable, and throwable.Table 13 compares affordance-recognition mAP across object datasets and novel classes.Source: Figure 7 · p. 13, H. Discussion with Prior Approaches · p. 13, Table 13 · p. 13
• Without a generic object detector, FCL's Unseen-category result drops to 0.Table 14 compares zero-shot detection results for FCL and ATL.Source: H. Discussion with Prior Approaches · p. 13, Table 14 · p. 13
• Fusing ATL and FCL prediction results substantially improves the combined result.Table 15 evaluates this complementarity using the released FCL model.Source: H. Discussion with Prior Approaches · p. 13, Table 15 · p. 13