Source-linked AI summary
Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer
Frederic Z. Zhang, Dylan Campbell, Stephen Gould
TL;DR
HOI detection has been dominated by transformer-based one-stage models, despite the training and memory demands of adapting DETR to interactiveness. This paper proposes a two-stage Unary-Pairwise Transformer that combines unary and pairwise representations, achieving state-of-the-art performance with lower training cost and 24 FPS inference using ResNet50. Its unary and pairwise layers respectively raise positive scores and suppress negative scores, while performance remains sensitive to object-detector confusion and other failure cases.
Problem
Transformer-based one-stage HOI detectors rely on difficult-to-train, memory-intensive decoders adapted from objectness to interactiveness, while earlier two-stage detectors lagged behind.
Method
The Unary-Pairwise Transformer is a two-stage HOI detector that refines DETR outputs with unary instance representations and pairwise human–object representations.
Results
The model outperforms state-of-the-art methods, and its ResNet50 version runs at 24 FPS on a single GeForce RTX 3090 device.
Takeaways & Limitations
Unary attention raises positive interaction scores while pairwise attention suppresses negative scores, producing complementary effects that boost performance.
Takeaways & Limitations
Object-detector confusion can propagate into action classification, and overlap or occlusion reduces confidence.
Abstract
from arXiv · showhide
Recent developments in transformer models for visual data have led to significant improvements in recognition and detection tasks. In particular, using learnable queries in place of region proposals has given rise to a new class of one-stage detection models, spearheaded by the Detection Transformer (DETR). Variations on this one-stage approach have since dominated human-object interaction (HOI) detection. However, the success of such one-stage HOI detectors can largely be attributed to the representation power of transformers. We discovered that when equipped with the same transformer, their two-stage counterparts can be more performant and memory-efficient, while taking a fraction of the time to train. In this work, we propose the Unary-Pairwise Transformer, a two-stage detector that exploits unary and pairwise representations for HOIs. We observe that the unary and pairwise parts of our transformer network specialise, with the former preferentially increasing the scores of positive examples and the latter decreasing the scores of negative examples. We evaluate our method on the HICO-DET and V-COCO datasets, and significantly outperform state-of-the-art approaches. At inference time, our model with ResNet50 approaches real-time performance on a single GPU.
1. Introduction
HOI detectors evolved from proposal-based two-stage systems to transformer-based one-stage models, but two-stage designs can recover strong performance when paired with DETR. The proposed model combines unary and pairwise representations to improve efficiency and interaction classification.
- Detection paradigms: HOI detection requires localising human–object pairs and classifying their actions, using either one-stage or two-stage architectures.Two-stage systems explicitly propose boxes before downstream interaction classification, whereas one-stage systems decode boxes and scores from shared representations.
- Motivation: Transformer-based one-stage HOI detectors often require substantial memory and difficult adaptation because DETR initially attends to objectness rather than interactiveness.The decoder must be adapted from object-focused attention to interaction-focused attention.
- Motivation: Replacing Faster R-CNN with DETR in the two-stage SCG detector significantly improved performance, bringing it on par with the state-of-the-art one-stage QPIC.The improvement is attributed to transformer representations and bipartite matching, which better align training labels with evaluation.
- Proposed approach: The proposed two-stage model refines DETR features using unary encodings of individual instances and pairwise encodings of human–object pairs.These representations provide complementary information for HOI classification.
- Contribution: The model is designed to outperform state-of-the-art methods while reducing training time and memory consumption.Lower resource use also permits more memory-intensive backbone networks.
2. Related work
Transformer-based detection uses patch tokens, attention, and learnable decoder queries to replace conventional region proposals. Related HOI systems extend DETR for joint human, object, and action prediction, while two-stage work explores feature decomposition, few-shot transfer, and modality fusion.
- Transformer foundations: Vision transformers represent images as spatial patch tokens and use attention to build flexible global representations.Learnable decoder queries are updated through self-attention and cross-attention with encoder tokens.
- Transformer foundations: DETR uses learnable object queries with bipartite matching to formulate object detection as set prediction without conventional region proposals.This design inspired many one-stage HOI detectors.
- One-stage HOI detection: Several one-stage HOI methods adapt DETR with additional heads or interaction decoders for human, object, and action prediction.Some methods also add action-related costs to bipartite matching.
- Two-stage HOI detection: Recent two-stage HOI research studies interaction decomposition, few-shot object representation, affordance transfer, and multimodal feature fusion.The proposed work uses modality fusion within its transformer model.
3. Unary–pairwise transformers
The interaction head uses unary and pairwise transformer layers to combine individual detections with human–object pair reasoning. It forms pairwise candidates, compares them, and combines action predictions with detector confidence scores for inference.
- Architecture: The interaction head uses a unary layer for individual human and object instances, followed by a pairwise layer for human–object pairs.The unary layer preferentially raises positive-pair scores, while the pairwise layer lowers negative-pair scores.
- Architecture: Pairwise positional encodings inject spatial information derived from bounding boxes into the modified transformer attention computation.The encodings capture unary box geometry and pairwise relations such as intersection-over-union, relative area, and direction.
- Token construction: The model retains DETR detections after non-maximum suppression and thresholding, then represents each detection with its box, confidence, class, and object-query feature.These detections provide the unary tokens and the inputs for computing pairwise positional encodings.
- Token construction: Pairwise tokens are formed from distinct pairs whose first token is human, fused with multi-branch fusion, and processed by an additional encoder layer before action classification.The pairwise encoder allows the network to compare HOI candidates.
- Training and inference: Final human–object scores combine detector confidence scores with sigmoid action logits, while focal loss and valid-action filtering address class imbalance and invalid combinations.During inference, λ > 1 suppresses overconfident objects, and invalid action–object combinations are assigned zero scores.
4. Experiments
Experiments show that the unary–pairwise model achieves strong HOI detection performance while using an efficient interaction head. Analyses attribute its behavior to complementary cooperative and competitive layers that raise positive scores and suppress hard negatives.
- Comparison with state-of-the-art methods: The proposed model achieves state-of-the-art HOI detection performance on HICO-DET and V-COCO, outperforming the next best method by a significant margin.Its performance remains strong across three backbone networks, with the gap widening for more powerful backbones.
- Efficiency: 24 FPS inference with ResNet50 on a single GeForce RTX 3090 demonstrates near-real-time operation.The detached backbone CNN and object-detection transformer also give the model a small memory footprint.
- Macroscopic effects of the interaction head: The cooperative layer increases positive-example scores by +0.15 on average while having little effect on negative examples.The analysis uses the same detected human–object pairs because the frozen object detector is unchanged across architectural variants.
- Macroscopic effects of the interaction head: The competitive layer decreases hard-negative scores by −0.11 on average, while slightly decreasing positive scores.The cooperative layer compensates for this positive-score decrease when both layers are used.
- Microscopic effects of the interaction head: Attention between unary tokens supports positive-pair scoring, with removing those weights decreasing scores by 0.06 (8%) on average.The intervention targets the three positive pairs and supports the proposed cooperative-layer mechanism.
- Microscopic effects of the interaction head: Pairwise attention behaves like soft non-maximum suppression by allowing dominant interactive pairs to suppress less promising pairs.Forcing the three interactive pairs to attend to one dominant pair decreased their scores by 0.08 (11%).
5. Conclusion
The paper proposes a two-stage HOI detector using unary and pairwise representations. Their complementary effects improve performance while reducing training time and memory consumption, with broader societal costs noted for transformer-based HOI systems.
- The proposed two-stage detector exploits unary and pairwise representations of human and object instances.
- Unary-token attention increases positive-example scores, whereas pairwise-token attention reduces negative-example scores like non-maximum suppression.
- The complementary unary and pairwise effects together boost performance significantly.
- Transformer models are computationally expensive and can impose a significant negative environmental impact.The paper also identifies potential misuse of HOI detection for unauthorised surveillance affecting minority and marginalised communities.
A. Pairwise positional encodings
Pairwise positional encodings combine unary box properties with pairwise geometry to represent human–object spatial relationships. The encoding uses concatenated features, positivity-preserving directional terms, and an MLP with numerical protection for logarithms.
- Each bounding-box pair is represented using unary features such as box areas and aspect ratios in addition to the box coordinates.
- Pairwise features include the box-area ratio, intersection over union, and directional encodings of the centres’ relative distance.
- Directional encodings are normalised by the first, human bounding box rather than by image dimensions.
- The function f(·) makes the feature vector componentwise positive before the complete encoding is computed by an MLP.
- A small constant ϵ is added to vectors before logarithms to avoid taking the logarithm of zero.
B. Numerical stability in the loss function
The loss formulation recovers a logit from the final interaction score because that score multiplies object confidence and action probability. A small constant supports numerical stability, while the supplied table caption specifies the comparison setting.
- The final interaction score is the product of a normalised object confidence score and an action probability.
- The corresponding logit is recovered before applying the loss because the model cannot directly use a logit loss on the multiplied final score.
- A small constant ϵ prevents taking the logarithm of zero during logit recovery.
- Table 5 compares cooperative-layer variants on the HICO-DET test set under the default setting using a ResNet50 backbone and one layer.
C. Multi-branch fusion
The multi-branch fusion module projects inputs into reduced-dimensional subspaces, fuses them within branches, restores the original size, and aggregates branches by elementwise summation.
- Each homogeneous branch maps two input features into a reduced-dimensional subspace and fuses them by elementwise product.
- The fused feature is mapped back to the original size, and elementwise summation aggregates the outputs across branches.
- The branch representation size is configured so the total number of parameters remains independent of the number of branches.
- The implementation uses linear-layer parameters, ReLU activation, and elementwise product operations.
D. Modified transformer encoder layer
The modified transformer encoder improves HOI reasoning by incorporating pairwise information into attention. Removing pairwise terms or using a vanilla encoder produces less informative attention patterns, while the full model identifies interactive correspondences more clearly.
- Encoder variants: 0.2 mAP is lost with a vanilla transformer encoder, and the gap remains after adding unary-box positional encodings.The comparison motivates directly incorporating pairwise terms into the cooperative layer.
- Attention behaviour: Pairwise positional encoding helps the modified encoder infer correspondences between interactive humans and objects.The attention maps show interactive instances attending to each other.
- Encoder variants: Without pairwise terms, the attention map becomes uniform along one dimension.This removes the structured correspondence pattern seen in the modified encoder.
- Attention behaviour: The vanilla encoder produces less interpretable attention, with no strong mutual attention between interactive instances and more attention between non-interactive ones.The modified encoder instead preferentially shares information between interactive instances.
- Attention behaviour: In crowded scenes, unary attention is symmetric between potentially interactive instances, while pairwise attention directs non-interactive pairs toward dominant pairs for suppression.These maps extend the observed cooperative and competitive behaviours to images with many instances.
E. Additional qualitative results
Additional visualisations show consistent HOI detection behaviour across complex scenes, while false positives reveal difficulty selecting the correct human when multiple candidates have plausible spatial relationships.
- Qualitative results: The model behaves consistently across different interaction types in scenes containing numerous human and object instances.Additional detected-HOI examples are provided for both HICO-DET and V-COCO.
- Failure cases: False positives on HICO-DET often arise when the model selects the wrong human instance for an interaction.The reported cause is reliance on plausible spatial relationships and human-instance saliency.
- Failure cases: Both human instances can appear plausible for boarding-airplane interactions when their spatial locations are similar.This example illustrates the ambiguity underlying one reported false-positive pattern.
F. Asset attrition
The supplied materials document qualitative-result figures, false-positive examples, dataset licensing information, and the public implementation of the modified encoder layer.
- Dataset assets: The HICO-DET annotations have no specified license, while its images are licensed under Creative Commons from Flickr.The licensing statement concerns annotations and images separately.
- Dataset assets: V-COCO annotations use the MIT License, while its images and annotations derive from MS COCO licensing and Flickr sources.MS-COCO annotations are licensed under Creative Commons Attribution 4.0, and image licenses vary by Flickr source.
- Figures: Figures 12 and 13 provide additional qualitative detected-pair results for HICO-DET and V-COCO, respectively.The materials also identify Figure 14 as a collection of HICO-DET false positives.
- Implementation: Listing 2 provides a PyTorch implementation of the modified transformer encoder layer, omitting the publicly available feedforward network.The implementation defines the encoder class, constructor, forward method, unary tokens, and pairwise positional encodings.