PaperlayerView exact revision on arXiv

Source-linked AI summary

Pairwise Body-Part Attention for Recognizing Human-Object Interactions

Hao-Shu Fang, Jinkun Cao, Yu-Wing Tai, Cewu Lu

arXiv:1807.10889v1cs.CV

TL;DR

HOI recognition methods often treat the body holistically, despite interactions relying on particular parts and their correlations. This paper combines pairwise body-part feature encoding with attention to select discriminative pairs, reporting a 10% relative mAP improvement over previous methods on HICO.

  • Problem

    Conventional HOI methods consider the body holistically, although interactions can rely on informative parts and correlations between them.

    Source: §1 · pp. 1–2

  • Method

    The model uses pairwise ROI pooling to encode body-part pairs and attention to select discriminative pair features for HOI prediction.

    Source: §1 · p. 2, ROI-pairwise Pooling · p. 6, Attention Module · p. 7

  • Results

    10% relative improvement in mAP over previous methods is reported on HICO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach improves recognition accuracy especially for HOI labels requiring detailed body-part information.

    Source: §5 · p. 15

    The model does not represent interactions of multiple people with the same object and can produce errors with ambiguous backgrounds or difficult detections.

    Source: Failure cases · p. 14

Abstract

from arXiv · show

In human-object interactions (HOI) recognition, conventional methods consider the human body as a whole and pay a uniform attention to the entire body region. They ignore the fact that normally, human interacts with an object by using some parts of the body. In this paper, we argue that different body parts should be paid with different attention in HOI recognition, and the correlations between different body parts should be further considered. This is because our body parts always work collaboratively. We propose a new pairwise body-part attention model which can learn to focus on crucial parts, and their correlations for HOI recognition. A novel attention based feature selection method and a feature representation scheme that can capture pairwise correlations between body parts are introduced in the model. Our proposed approach achieved 4% improvement over the state-of-the-art results in HOI recognition on the HICO dataset. We will make our model and source codes publicly available.

1 Introduction

The introduction argues that HOI recognition should move beyond holistic body features because relevant gestures vary while informative body parts and their relationships matter. It proposes pairwise body-part attention and reports improved HICO performance over prior methods.

1 Introduction

  • Conventional HOI methods use holistic or coarse body representations, while attention to informative parts varies with context.For same-label mug-holding examples, gestures differ while the arm holding the mug remains shared.Source: §1 · pp. 1–2
  • Pairwise ROI pooling represents joint features of body-part pairs, and attention selects discriminative correlations while filtering irrelevant body gestures.The end-to-end loss guides selection of representative pairwise features for each HOI label.Source: §1 · p. 2
  • The paper presents its work as the first use of attention on human body-part correlations for HOI recognition.It further reports state-of-the-art results on HICO and MPII.Source: §1 · p. 2
  • 10% relative improvement in mAP over previous methods on HICO is reported for the proposed approach.The authors evaluate on HICO and MPII and describe their result as state of the art.Source: §1 · p. 2

2 Related work

Prior work addresses HOI through contextual cues, body parts, whole-person features, and attention methods. This paper distinguishes its approach by explicitly modeling pairwise body-part correlations and learning attention over those pairs.

2 Related work

  • Still-image pose can be ambiguous without motion cues, and high-level context has improved action-recognition performance in HOI settings.Other approaches also use discriminative image patches, attributes, BoF, color, or semantic hierarchy.Source: §2 · p. 3
  • Deep HOI methods have used coarse body-part boxes or fused person-box and whole-image features.Source: §2 · p. 3
  • This method models body parts and pairwise correlations, unlike Gkioxari et al.’s coarse parts and Mallya et al.’s whole-person boxes.It also learns attention over body parts and their spatial configurations.Source: §2 · p. 3
  • The framework pools pairwise part features, selects discriminative pairs with attention, and combines them with global human, object, and scene features.It adopts MIL to address multi-person co-occurrence [29].Source: Figure 2 · p. 4
  • Prior attention approaches focus on informative video joints or whole-image pooling, rather than explicit body-part correlations.Source: §2 · p. 4

3 Our Method

The method combines global appearance cues with local pairwise body-part features to recognize human-object interactions. It uses attention to select and weight informative body-part pairs before combining them with global features for prediction.

3 Our Method

  • The model combines global appearance features with local pairwise body-part features, then uses separate final fully connected layers to predict interactions for each detected person.The global features include person, scene, and object information.Source: §3 · p. 4, Incorporating Object Features · p. 5, §3.3 · p. 8

Scene and Human Features

  • The base network uses person and scene features, with multiple instance learning taking the maximum HOI score across person predictions in an image.The method further incorporates object features into this base network.Source: §3 · p. 4, Scene and Human Features · p. 5

Incorporating Object Features

  • Object features are incorporated by pooling the union of the detected person and object boxes, which represents their relative locations.For multiple objects around a person, the method samples multiple person-object union boxes and pools each one.Source: Incorporating Object Features · p. 5

3.2 Local Pairwise Body-part Features

  • ROI-pairwise pooling extracts joint features for a body-part pair while preserving its relative spatial arrangement and suppressing irrelevant regions between parts.It zeros activations outside the two part boxes, then max-pools the union-box feature map to a fixed size.Source: ROI-pairwise Pooling · p. 6

Attention Module

  • The pairwise attention module estimates an importance score for each pooled body-part pair, keeps the top k pairs, and scales their feature maps by their scores.The selected pair features are trained end-to-end; the attention scores require no human supervision.Source: Figure 3 · p. 6, Attention Module · p. 7

4 Experiment

The experiments use HICO and MPII, with dataset-specific training and evaluation setups. The section describes the datasets, sampled validation sets, and baseline and loss settings.

4.1 Setting

  • Experiments evaluate the model on HICO and MPII, which differ in label structure and whether an image may contain multiple people performing different actions.HICO has 600 HOI labels and image-level labels; MPII classifies each image into one of 393 action classes.Source: Dataset · p. 8

Dataset

  • HICO has 38,116 training images and 9,658 test images, with 10,000 training images sampled for validation.MPII has 15,205 training images and 5,708 test images; 6,987 training images are sampled for validation.Source: Dataset · p. 8

HICO

  • For HICO, the setup uses Faster R-CNN human and object proposals, defines 10 keypoint-based body parts, and samples 45 distinct body-part pairs.The training setup follows, and uses weighted sigmoid cross-entropy with positive and negative weights of 10 and 1.Source: HICO · pp. 8–9
  • The reproduced Mallya and Lazebnik baseline reaches 35.6 mAP, compared with its reported 36.1 mAP.For MPII, the model is trained per person without MIL and without weighted loss.Source: HICO · pp. 8–9, MPII · p. 9

MPII

On MPII, the VGG16-based model achieves 36.8 mAP, and using the same base model it outperforms by 1.4 mAP. Qualitative results visualize selected body-part pairs alongside predicted interactions.

  • The qualitative examples include predictions for interactions involving bicycles, horses, backpacks, kites, ties, and snowboards.Source: §4.2 · p. 10
  • 36.8 mAP on MPII is achieved by the VGG16-based model, which also outperforms by 1.4 mAP using the same base model.The paper reports results on the MPII test set in Table 2.Source: Table 2 · p. 10, §4.2 · p. 11
  • Red boxes show the body-part pairs with the highest attention scores in qualitative predictions.The examples include HICO and MPII predictions, with detected human boxes in green and predicted HOIs underneath.Source: §4.2 · pp. 10–11

4.3 Ablative studies

The HICO ablations compare object-feature and object-box choices in the baseline. Adding object features and using union-box features both improve mAP, and the improved baseline exceeds the result reported by.

Improvements from body parts information

Adding body-part features improves the baseline, and pairwise body-part features with attention achieve the strongest reported result in these ablations. Combining single-part and pairwise attention candidates performs slightly worse.

Improvements from body parts information

Pairwise Body-part Attention

  • Concatenating features from 45 body-part pairs with global appearance features achieves 38.9 mAP.The authors report that exploiting spatial relationships between body parts benefits HOI recognition.Source: Pairwise Body-part Attention · p. 12
  • 39.9 mAP is achieved when attention selects pairwise body-part features with k set to 20.The fully connected layers receive information from fewer selected parts.Source: Pairwise Body-part Attention · p. 12
  • 39.1 mAP results from jointly selecting single body parts and body-part pairs, slightly below the pairwise-attention result.The authors suggest local single-part features may already be encoded by ROI-pairwise pooling, so extra body-part information may distract attention.Source: Pairwise Body-part Attention · p. 12

4.4 Analysis

The analysis finds that selecting 20 pairwise feature maps gives the best validation performance among tested values. A human judgment study assesses whether selected pairs match HOI labels, and failure examples identify detector and background issues.

Parameter for Feature Selection Layer

  • Validation performance rises as k increases to 20, then drops; using all 45 pairs is 1.2 mAP below the highest accuracy.At k = 45, the feature-selection layer is not used, supporting the authors’ conclusion that rejecting irrelevant pairs matters.Source: Parameter for Feature Selection Layer · p. 12, Figure 5 · p. 13

Evaluation of Attention

  • Failure examples include confusion from ambiguous background features and failures when the human or object detectors fail.Source: Figure 6 · p. 13
  • Human judgments rate selected body-part pairs at 0.28 top-1 accuracy and 0.76 top-5 accuracy for relevance to HOI labels.A pair counts as correct when at least half of 30 participants judge it important for the HOI label.Source: Evaluation of Attention · pp. 12–13

Improvements by HOI class

The model improves recognition for HOIs requiring detailed body-part information, while failure cases include ambiguous backgrounds, unmodeled multi-person interactions, and difficult detections.

Improvements by HOI class

  • The model shows great improvement over the baseline for detailed-part HOIs such as surfboard holding, apple buying, and bird releasing.Table 5 compares mAP for 20 randomly selected HICO categories against Mallya and Lazebnik [29].Source: Improvements by HOI class · p. 13, Table 5 · p. 14
  • Table 4 lists the most-selected body-part pairs for HOIs, with “l” and “r” marking left and right.Source: Table 4 · p. 14

Failure cases

  • Errors include background-driven predictions, unmodeled interactions between multiple people and one object, missed detections, and ambiguous scenes.Background features can prompt a “washcar” label, while two men on one motorcycle may be classified as racing motorcycle.Source: Failure cases · p. 14

5 Conclusions

The paper proposes pairwise body-part attention using ROI-pairwise pooling and combines selected pairwise features with background, human, and object features for HOI prediction. It reports improved accuracy, especially for detailed-part HOIs, and identifies multi-person interactions as future work.

5 Conclusions

  • ROI-pairwise pooling and pairwise body-part attention select pair features that are combined with background, human, and object features for HOI prediction.Source: §5 · p. 14
  • The approach is robust and significantly improves recognition accuracy, especially for HOI labels requiring detailed body-part information.Source: §5 · p. 15
  • Future work will investigate incorporating multi-person interactions into HOI recognition.Source: §5 · p. 15
Loading 1807.10889v1…