PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Compositional Learning for Human-Object Interaction Detection

Zhi Hou, Xiaojiang Peng, Yu Qiao, Dacheng Tao

arXiv:2007.12407v2cs.CV

TL;DR

HOI detection faces a long-tailed space of verb-object combinations, making rare interactions difficult to learn. VCL decomposes HOI representations into verb and object features and composes new interaction samples from them. It improves HOI generalization on HICO-DET and V-COCO and outperforms recent state-of-the-art methods on HICO-DET.

  • Problem

    HOI detection must localize humans and objects and infer their relationships, but possible verb-object combinations have a long-tailed distribution with many rare training categories.

  • Method

    VCL decomposes HOI representations into verb and object-specific features, shares them across images and HOI types, and composes novel interaction samples in feature space.

  • Results

    VCL improves HOI detection generalization on HICO-DET and V-COCO and outperforms recent state-of-the-art methods on HICO-DET.

  • Takeaways & Limitations

    Visual composition provides a framework for addressing long-tail, low-shot, and zero-shot HOI detection through shared verb and object representations.

Abstract

from arXiv · show

Human-Object interaction (HOI) detection aims to localize and infer relationships between human and objects in an image. It is challenging because an enormous number of possible combinations of objects and verbs types forms a long-tail distribution. We devise a deep Visual Compositional Learning (VCL) framework, which is a simple yet efficient framework to effectively address this problem. VCL first decomposes an HOI representation into object and verb specific features, and then composes new interaction samples in the feature space via stitching the decomposed features. The integration of decomposition and composition enables VCL to share object and verb features among different HOI samples and images, and to generate new interaction samples and new types of HOI, and thus largely alleviates the long-tail distribution problem and benefits low-shot or zero-shot HOI detection. Extensive experiments demonstrate that the proposed VCL can effectively improve the generalization of HOI detection on HICO-DET and V-COCO and outperforms the recent state-of-the-art methods on HICO-DET. Code is available at https://github.com/zhihou7/VCL.

1 Introduction

HOI detection must handle complex, long-tailed verb-object combinations, motivating VCL’s decomposition of interactions into shared verb and object representations and their recombination into novel HOIs.

  • 1 Introduction: HOI detection localizes humans and objects while inferring their verb-object relationships, but rare interaction types often lack sufficient training samples.A few interactions dominate the data, whereas many combinations are rare.
  • 1 Introduction: Decomposing HOIs into verb and object components enables feature sharing across interactions; for example, ⟨hug, suitcase⟩ can combine abundant suitcase and hug evidence.The cited example contrasts one ⟨hug, suitcase⟩ example with more than 1000 suitcase and 500 hug samples.
  • 1 Introduction: VCL learns shared visual verb and object representations across images and HOI types, then composes features such as ⟨ride, horse⟩ from ⟨feed, horse⟩ and ⟨ride, bicycle⟩.This composition uses semantic horse and ride features from separate interactions.
  • 1 Introduction: VCL addresses verb-feature entanglement by extracting verb representations from the human-object union box and sharing weights between verb and human streams.The paper states that this produces more discriminative cues for final detection.
  • 1 Introduction: VCL composes novel HOI samples from decomposed verbs and objects to relieve low-shot and zero-shot detection issues.The contribution also reports that VCL considerably outperforms previous state-of-the-art methods.

2 Related Works

Prior HOI work improves feature modeling with streams, attention, interactiveness, graphs, and pose, while VCL targets long-tail and zero-shot recognition through visual composition.

  • 2 Related Works: HOI detection is framed as human-centric relation detection and is supported by datasets including V-COCO and HICO-DET.The related-work discussion distinguishes HOI from visual relationship detection.
  • 2 Related Works: Earlier methods combine visual and spatial features, attention, interactiveness, contextual modeling, graph neural networks, or pose-aware features to improve HOI detection.The cited approaches include multi-stream, instance-centric attention, contextual attention, graph, and pose-aware models.
  • 2 Related Works: Factorized HOI modeling also separates verb and object reasoning for long-tail recognition, whereas VCL composes HOI examples from these components.The related-work passage identifies this as similar in motivation but different in approach.
  • 2 Related Works: Other compositional methods decompose scenes or synthesize features, but Compositional GAN and LaSO compose whole-image features and rely on generation or reconstruction losses.VCL instead performs composition on decomposed verb and object representations.
  • 2 Related Works: The VCL framework overview uses shared parameters across images and classifies composited and verb-object representations with a shared fully connected classifier.Its spatial-human branch uses an individual classifier.

3 Visual Compositional Learning

VCL detects HOIs through spatial-human and verb-object branches, while a composition branch synthesizes feature-space interactions within and between images to improve rare and unseen HOI generalization.

  • Overview: VCL uses a randomly selected image pair, detects humans and objects, and extracts human, verb, and object features with ROI-Pooling and Residual CNN blocks.
  • Multi-branch Network: The framework combines spatial-human, verb-object, and composition branches, with the composition branch training HOI detection on synthesized interactions.
  • Multi-branch Network: The verb-object branch extracts verb representations from human-object union boxes and shares weights with the spatial-human branch's human stream.
  • Composing Interactions: VCL composes new verb-object samples within and between images by generating possible pairs, removing infeasible label combinations, and filtering them.
  • Composing Interactions: The HOI label space is decoupled into verb-HOI and object-HOI cooccurrence matrices, enabling binary HOI labels to produce verb and object label vectors for composition.
  • Training and Inference: During training, cross-entropy losses supervise original spatial-human and verb-object instances alongside composited interactions, with composition applied in each minibatch.

4 Experiment

Experiments on HICO-DET and V-COCO evaluate VCL against state-of-the-art methods, zero-shot detection, and component variants. VCL improves rare and unseen interaction detection, with gains linked to compositional learning, verb representation, branch design, and composing across images.

  • Comparisons with state-of-the-art: 19.43% and 22.00% mAP were achieved on HICO-DET Full under Default and Known Object settings without external knowledge.Against prior methods, VCL improved Full mAP by 1.97% and Rare mAP by 0.9% over the previous Rare-category leader.
  • Comparisons with state-of-the-art: 23.63% mAP was obtained using a ResNet-101 backbone and a detector fine-tuned on HICO-DET.The comparison indicates that object-detector performance substantially affects this two-stage HOI detection setting.
  • Generalized Zero-shot HOI Detection: VCL consistently improved seen and unseen categories in zero-shot detection, with unseen-category gains exceeding 4% for both selection strategies.The baseline achieved 3.30% and 5.06% in the unseen setting while predicting all 600 classes.
  • Ablation Analysis: VCL increased HICO-DET Full performance by 1.00% and Rare performance by 2.41%, with a larger benefit for rare than non-rare categories.On V-COCO, removing VCL reduced performance by 0.8%, while VCL also improved discriminative action representation.
  • Ablation Analysis: Learning verb features from the human-object union box improved HICO-DET by 0.86% and V-COCO by 0.6% within VCL.Sharing weights between human and verb streams further raised HICO-DET mAP from 18.93% to 19.43%.
  • Ablation Analysis: Composing interactions within and between images reached 19.43% Full and 16.55% Rare mAP, whereas within-image composition alone was similar to baseline.Between-image composition alone reached 19.06% Full mAP.

5 Conclusion

The paper presents VCL as a compositional framework for alleviating long-tail and zero-shot HOI detection challenges, evaluated on two benchmarks.

  • VCL composes interactions from shared verb and object latent representations between and within images.
  • The framework extracts verb representations from the union box of human and object rather than the human box.
  • VCL is evaluated on two HOI detection benchmarks, with particular attention to low-shot and zero-shot detection.

A The two branches study in zero-shot HOI detection

The study compares verb-object and spatial-human branches for zero-shot HOI detection on HICO-DET. Their relative performance is similar in Seen and Full categories, while the verb-object branch performs better in two Unseen settings.

  • The ablation evaluates the two VCL branches during inference on the HICO-DET zero-shot HOI detection test set.
  • Verb-object and spatial-human branches achieve similar performance in the Seen and Full categories.
  • 3.52% and 4.90% gains favor the verb-object branch over the spatial branch when selecting rare first and non-rare first, respectively, in the Unseen category.

B The effect of the number of interactions in minibatch

The minibatch study increases interactions while reducing augmentations and negative interactions so VCL can still be optimized on a single GPU. Different interaction counts produce similar baseline results, but the larger improvement is obtained under one setting.

  • VCL increases interactions per minibatch while reducing augmentations and negative interactions per interaction to preserve single-GPU optimization.
  • 18.43 mAP and 18.47 mAP are reported for baseline models with different numbers of interactions.
  • 1.0 mAP versus 0.44 mAP indicates a larger improvement for one interaction-count setting than the other.

C Hyper-Parameters and Baseline

This section reports hyper-parameter evaluation and baseline detection settings. Testing uses detector-specific score thresholds, while reweighting and detection postprocessing are identified as improvement strategies for the baseline.

  • Hyper-Parameters: The proposed framework contains two hyper-parameters, λ1 and λ2, whose values are evaluated experimentally.
  • Baseline: At test time, the method first detects image objects and then infers HOI categories from the detection results.
  • Baseline: The ResNet-50 COCO detector uses score thresholds of 0.8 for humans and 0.3 for objects.
  • Baseline: The fine-tuned ResNet-101 detector uses thresholds of 0.3 for humans and 0.1 for objects because its object detections are reported as substantially better.
  • Baseline: Reweighting assigns class-dependent cross-entropy weights, while detection postprocessing lowers thresholds when no objects or humans are detected.

D Unseen labels

The released code provides the labels for two selection strategies.

  • The authors released code containing labels for two selection strategies.
  • The labels identify the paper’s two selection strategies.
  • Readers can consult the released code to find these selection-strategy labels.
Loading 2007.12407v2…