PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Context-aware Interaction Recognition

Bohan Zhuang, Lingqiao Liu, Chunhua Shen, Ian Reid

arXiv:1703.06246v3cs.CV

TL;DR

Interaction recognition must account for the fact that the same interaction can occur with different object contexts, while context-independent classifiers can perform poorly. This paper makes interaction classifiers adaptive to context using semantic representations, and reports state-of-the-art performance on two visual relationship detection datasets.

  • Problem

    The same interaction can involve different object pairs, and context-independent interaction classifiers can perform poorly because associating interactions with visual appearance without context is difficult.

    Source: §1 · p. 2

  • Method

    The framework represents subject-object context in a semantic space and uses it to adapt one classifier per interaction type.

    Source: §3.1 · pp. 3–4

  • Results

    Experiments report superior performance over competing methods, with state-of-the-art performance on two visual relationship detection datasets.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The framework is reported to generalize to unseen context-interaction combinations, with reasonable predictions on most illustrated unseen examples.

    Source: §4.1.2 · p. 8, §5 · p. 9

    The method assumes subject and object contexts have already been detected, with bounding boxes and labels given, and classifies their interaction type.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Recognizing how objects interact with each other is a crucial task in visual recognition. If we define the context of the interaction to be the objects involved, then most current methods can be categorized as either: (i) training a single classifier on the combination of the interaction and its context; or (ii) aiming to recognize the interaction independently of its explicit context. Both methods suffer limitations: the former scales poorly with the number of combinations and fails to generalize to unseen combinations, while the latter often leads to poor interaction recognition performance due to the difficulty of designing a context-independent interaction classifier. To mitigate those drawbacks, this paper proposes an alternative, context-aware interaction recognition framework. The key to our method is to explicitly construct an interaction classifier which combines the context, and the interaction. The context is encoded via word2vec into a semantic space, and is used to derive a classification result for the interaction. The proposed method still builds one classifier for one interaction (as per type (ii) above), but the classifier built is adaptive to context via weights which are context dependent. The benefit of using the semantic space is that it naturally leads to zero-shot generalizations in which semantically similar contexts (subjectobject pairs) can be recognized as suitable contexts for an interaction, even if they were not observed in the training set.

1. Introduction

Interaction recognition is difficult because the same interaction can involve visually dissimilar object pairs, making context informative. The paper proposes context-adaptive classifiers to address limits of both combination-based and context-independent approaches, and reports improved performance.

1. Introduction

  • The same interaction can look visually dissimilar across object pairs, as in cows eating grass versus people eating bread, making subject and object context potentially important for recognition..Source: §1 · p. 2
  • Treating each interaction-context combination as a class scales poorly and cannot generalize to unseen combinations, even when the interaction and context were each seen during training.This approach requires training images for potentially huge numbers of combinations.Source: §1 · p. 2
  • Classifying interactions independently of context can perform poorly because associating interactions with visual appearance without context is difficult.Language priors and structural learning have been used to avoid unreasonable interaction-context combinations, but still rely on context-independent interaction classifiers.Source: §1 · p. 2
  • The proposed framework keeps one classifier per interaction but adapts its parameters to context, encouraging similar contexts to have similar classifiers without assigning each combination its own class.The paper reports improvements across investigated feature representations, with an attention mechanism yielding further gains and the best-performing model.Source: §1 · p. 2
  • The framework and attention-augmented model achieve superior performance over competing methods in the reported experiments.The paper reports improvements for all investigated feature representations and further improvements from attention.Source: §1 · p. 2

2. Related work

Related work spans action recognition with limited context coverage, visual-relationship detection, and language-guided visual recognition. Figure 1 contrasts two interaction-classification extremes with the paper’s context-adaptive approach.

2. Related work

  • Action-recognition datasets often cover interactions with limited context; even the relatively large HICO dataset has 600 human-object interaction categories.Source: §2 · p. 2
  • Visual-relationship work has improved detection and scalability, using language priors or end-to-end detectors that reason over relationship components.Source: §2 · p. 2
  • Figure 1 contrasts treating each interaction-context combination as a class with context-independent interaction classification; the proposed approach uses one context-adaptive classifier per interaction.Source: Figure 1 · p. 3
  • Language information is also used in phrase grounding [24] [11], visual question answering, and one-shot learning.Phrase-grounding works [11] employ attention models to extract linguistic cues from phrases.Source: §2 · p. 3

3. Methods

The framework retains one classifier per interaction but makes its parameters context-dependent, using semantic subject-object representations to generate auxiliary weights. It explores spatial and appearance features, then attention-based pooling, including attention conditioned on the interaction and context.

3.1. Context-aware interaction classification framework

  • With detected subject and object boxes and labels, each interaction gets a classifier combining a context-independent term with a semantic-context-derived auxiliary term.The subject-object word2vec embeddings are projected into a semantic space to generate the auxiliary classifier weights.Source: §3.1 · pp. 3–4

3.2. Feature representations for interactions recognition

  • The paper investigates spatial and appearance feature representations for characterizing interactions within the context-aware classification framework.The feature representation is treated as an important design choice for interaction prediction.Source: §3.2 · p. 4

3.2.1 Spatial feature representation

  • Spatial features combine subject-box, object-box, and pairwise box geometry into a 14-dimensional representation passed to the context-aware classifier.Source: §3.2.1 · p. 4

3.2.2 Appearance feature representation

  • Appearance features come from mean-pooled VGG-16 conv5_3 activations on the resized union of subject and object boxes, fine-tuned end-to-end with the classifier.Source: §3.2.2 · p. 4

3.3. Improving appearance representation with attention and context-aware attention

  • Attention pooling weights convolutional locations to emphasize discriminative visual regions; the proposed generator conditions attention weights on the interaction and subject-object context.The resulting interaction-specific attention-pooled vector is sent to its corresponding context-aware classifier.Source: §3.3 · pp. 4–5

3.4. Implementation details

  • Models use cross-entropy loss and Adam; appearance models initialize VGG-16 from ImageNet and use learning rates 0.001 for new layers and 0.0001 for VGG-16 layers.Source: §3.4 · pp. 5–6

4. Experiments

The experiments compare eight interaction-recognition methods on the Visual Relationship Detection and Visual Phrase datasets. They evaluate standard detection and zero-shot settings, with context-aware methods outperforming the reported baselines and competing methods.

4. Experiments

  • Eight methods are compared, including context-independent interaction and joint interaction-context baselines alongside context-aware classifiers.The compared variants include appearance and spatial features, with attention-pooling representations among the context-aware methods.Source: §4 · p. 6
  • Comparisons with published methods are qualified because they may use different feature representations, detectors, or pretraining strategies.Source: §4 · p. 6

4.1. Evaluation on the Visual Relationship dataset

  • VRD evaluation follows across predicate, phrase, and relationship detection; phrase and relationship detection use object detections, and performance is measured by Recall@50 and Recall@100.Phrase detection localizes the entire relationship in one box; relationship detection localizes subject and object boxes, both of which must overlap ground-truth boxes by at least 0.5.Source: §4.1 · p. 7

4.1.1 Detection results comparison

  • Across VRD, context-aware AP methods outperform four modeling baselines; AP+C+CAT also beats and, with results most comparable to.The comparison with uses the same object detection, and the reported improvement comes without explicit language priors.Source: §4.1.1 · p. 7, Table 1 · p. 8

4.1.2 Zero-shot learning performance evaluation

  • On 1,877 unseen VRD relationships, AP+C+CAT almost doubles predicate-detection performance versus Language Priors.The authors attribute the gain to context-aware modeling of visual appearance variations across contexts.Source: §4.1.2 · pp. 7–8, Table 3 · p. 9
  • Zero-shot VRD results favor spatial over appearance features; the authors speculate that spatial features are more object-independent and less prone to overfitting.This contrasts with the standard VRD results, where appearance features perform consistently better, especially in the baseline setting.Source: §4.1.1 · p. 7, §4.1.2 · p. 8

4.1.3 Extensions and comparison with the state-of-the-art methods

  • With Faster-RCNN proposals, AP+C+CAT performs best on phrase detection R@50, relationship detection, and zero-shot phrase and relationship detection; language priors add only marginal improvement.The detector is replaced with Faster-RCNN without end-to-end joint training.Source: Table 2 · p. 8, §4.1.3 · p. 9

4.2. Evaluation on the Visual Phrase dataset

  • On Visual Phrase, AP+C+CAT again achieves the best performance, with its largest improvement over in zero-shot learning.The evaluation uses the dataset’s setting from and RCNN-generated candidate regions and detection scores.Source: §4.2 · p. 9

5. Conclusion

The paper concludes that its context-aware interaction framework is accurate, scalable, and able to generalize to unseen context–interaction combinations. Experiments validate its methods and report state-of-the-art performance on two visual relationship detection datasets.

5. Conclusion

  • The proposed methods achieve state-of-the-art performance on two visual relationship detection datasets.The paper reports extensive experiments validating their advantage.Source: §5 · p. 9
  • The framework is designed to recognize unseen context–interaction combinations while remaining accurate and scalable.Source: §5 · p. 9
  • The paper investigates visual representations for interaction patterns and extends the framework with an attention-pooling layer.Source: §5 · p. 9
  • Figures 4 and 5 show interaction predictions against ground truth while using ground-truth context boxes.Figure 4 shows interaction recognition examples; Figure 5 shows zero-shot examples.Source: Figure 4 · p. 10, Figure 5 · p. 10
  • Table 4 presents a performance comparison on the Visual Phrase dataset.Source: Table 4 · p. 11

Related papers

Loading 1703.06246v3…