Source-linked AI summary
Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, Jing Shao, Chen Change Loy
TL;DR
Visual relationship recognition must identify complex ⟨subject-predicate-object⟩ interactions amid diverse layouts and ambiguous, imbalanced annotations. Zoom-Net addresses this with feature-level interactions through stacked SCA-M pooling cells and IH-trees, without linguistic priors or label-level interaction modeling. The paper reports state-of-the-art performance on Visual Genome and VRD, while showing benefits when its components are transferred to existing approaches.
Problem
Visual relationship recognition must recognize ⟨subject-predicate-object⟩ tuples across diverse spatial layouts, while Visual Genome contains ambiguous labels and extremely imbalanced annotations.
Method
Zoom-Net stacks SCA-M modules for bidirectional object-predicate feature interaction and uses IH-trees for hierarchical recognition of ambiguous and imbalanced labels.
Results
Zoom-Net achieves state-of-the-art performance on both Visual Genome and VRD, and SCA-M and IH-trees improve existing approaches and recognition robustness.
Takeaways & Limitations
Feature-level interactions can achieve state-of-the-art visual relationship recognition without explicit or implicit label-level interactions.
Takeaways & Limitations
The evaluation prunes extremely rare labels with fewer than 10 samples for valid evaluation.
Abstract
from arXiv · showhide
Recognizing visual relationships <subject-predicate-object> among any pair of localized objects is pivotal for image understanding. Previous studies have shown remarkable progress in exploiting linguistic priors or external textual information to improve the performance. In this work, we investigate an orthogonal perspective based on feature interactions. We show that by encouraging deep message propagation and interactions between local object features and global predicate features, one can achieve compelling performance in recognizing complex relationships without using any linguistic priors. To this end, we present two new pooling cells to encourage feature interactions: (i) Contrastive ROI Pooling Cell, which has a unique deROI pooling that inversely pools local object features to the corresponding area of global predicate features. (ii) Pyramid ROI Pooling Cell, which broadcasts global predicate features to reinforce local object features.The two cells constitute a Spatiality-Context-Appearance Module (SCA-M), which can be further stacked consecutively to form our final Zoom-Net.We further shed light on how one could resolve ambiguous and noisy object and predicate annotations by Intra-Hierarchical trees (IH-tree). Extensive experiments conducted on Visual Genome dataset demonstrate the effectiveness of our feature-oriented approach compared to state-of-the-art methods (Acc@1 11.42% from 8.16%) that depend on explicit modeling of linguistic interactions. We further show that SCA-M can be incorporated seamlessly into existing approaches to improve the performance by a large margin. The source code will be released on https://github.com/gjyin91/ZoomNet.
1 Introduction
Visual relationship recognition must handle diverse spatial layouts and complex interactions between localized objects. Zoom-Net addresses this through feature-level message sharing, SCA-M pooling cells, and IH-trees for ambiguous and imbalanced annotations, without explicit label-level interactions.
- Visual relationship recognition predicts ⟨subject-predicate-object⟩ tuples despite diverse spatial layouts and complex inter-object interactions.
- Zoom-Net learns spatiality-aware contextual features by propagating object features to predicate features and broadcasting predicate features back to subject and object features.
- SCA-M combines Contrastive ROI Pooling, which deROI-pools object features into predicate context, with Pyramid ROI Pooling, which broadcasts predicate features to object regions.
- Stacked SCA-M modules enable multi-scale bidirectional message passing that improves individual object recognition and inter-object relation learning.
- IH-trees reformulate one-hot classification as n-hot hierarchical recognition to mitigate ambiguity and severe imbalance in Visual Genome annotations.
- Zoom-Net uses feature-level interactions alone rather than explicit or implicit label-level interactions, while SCA-M and IH-trees can also benefit existing label-interaction methods.
2 Related work
Prior work applies contextual learning and class hierarchies in related vision tasks, but contextual feature learning for visual relationship recognition has received limited attention.
- Contextual Learning: Contextual information has been used in object detection, segmentation, and retrieval by combining localized detector features with global visual features.
- Class Hierarchy: Class hierarchies encode label relations or abstraction levels to improve classification and retrieval performance.
- Contextual Learning: Contextual feature learning for visual relationship recognition remains little explored in previous work.
3 Zoom-Net: Mining Deep Feature Interactions
Zoom-Net mines feature-level interactions among subject, predicate, and object branches through stacked Spatiality-Context-Appearance Modules, combining spatiality-aware local-global context with hierarchical classifiers. Its pooling cells enable bidirectional, multi-scale message passing between object and predicate features.
- Motivation: Zoom-Net extends visual relationship recognition beyond linguistic priors by mining feature-level interactions among subject, predicate, and object branches.The model is designed as an end-to-end feature-interaction approach rather than explicit linguistic interaction modeling.
- Appearance, Context and Spatiality: SCA-M combines an Appearance Module, a Context-Appearance Module, and spatiality-aware local-global contextual feature integration.Appearance processing is independent, direct pairwise fusion provides context, and SCA-M integrates local and global context spatially.
- SCA-M: Contrastive ROI Pooling uses ROI and deROI operations to map subject or object features back onto the predicate feature’s spatial palette.The deROI operation creates spatiality-aware object features aligned with the predicate feature map, while regions outside the object ROI are zeroed.
- SCA-M: Pyramid ROI Pooling broadcasts global predicate context into subject and object branches through multi-scale ROI operations.Together with Contrastive ROI Pooling, it supports deep interactions between local object features and global predicate features.
- Stacked SCA-M: Stacked SCA-Ms aggregate dynamic spatial and contextual information across branches before separate subject, predicate, and object classification.The architecture shares lower-layer feature extraction, applies branch-specific appearance convolutions, stacks two SCA-Ms, and then uses three classifiers.
4 Hierarchical Relational Classification
Visual Genome’s broad, ambiguous, and imbalanced label space motivates hierarchical classification. Zoom-Net uses intra-hierarchical trees to cluster related object and predicate labels and improve raw-label prediction.
- IH-trees group rare, noisy, and ambiguous labels with clearly defined labels using intra-class correlations for objects and predicates.The approach avoids modeling cross-triplet label interactions through external linguistic priors.
- Object IH-tree: The object IH-tree normalizes noisy noun labels, clusters semantically related concepts, and represents them across three hierarchical levels.Examples include reducing variants such as “old man,” “bald man,” and “men” to “man,” then separating concepts such as clothing, animals, and vehicles.
- Hierarchical objectives: Hierarchical outputs use separate softmax vectors and losses across levels, encouraging intra-level label exclusion and inter-level label dependency.The losses are combined for subject, predicate, and object branches through α, β, and γ.
- Predicate IH-tree: The predicate IH-tree normalizes keywords and separately organizes verb and preposition structures because their contextual representations differ.Phrases such as “wears a,” “wearing a yellow,” and “wearing a pink” map to “wear,” while verb-preposition phrases can participate in separate partitions.
5 Experiments on Visual Genome (VG) Dataset
Experiments on Visual Genome evaluate Zoom-Net through recognition metrics, SCA-module ablations, IH-tree ablations, and comparisons with prior methods. The results attribute gains to contextual and spatial feature interactions and hierarchical label structure.
- Evaluation setup: Visual Genome evaluation reports Acc@1, Acc@5, Rec@50, and Rec@100 for subject, predicate, object, and relationship recognition.Relationship accuracy is averaged across subject, predicate, and object accuracies, while recall addresses incomplete annotations.
- SCA-M ablation: 12.25% is the Top-1 predicate-accuracy gap between vanilla A-M and SCA-M, compared with a 4.96% subject-accuracy gap.The larger predicate gain is consistent with the module’s focus on interactions and contextual cues.
- SCA-M ablation: Contextual information improves subject, predicate, object, and relationship recognition: both CA-M and SCA-M significantly outperform vanilla A-M.The SCA-M additionally captures correlations between spatiality and semantic appearance cues across objects.
- IH-tree ablation: Removing IH-trees causes performance drops of around 1%–4% across different metrics, while hierarchical labels improve raw object, predicate, and relationship recognition.The hierarchy also brings semantically related alternatives such as “wearing” and “wears” into top predictions.
- State-of-the-art comparison: Zoom-Net significantly outperforms DR-Net and ViP on Visual Genome, which the analysis associates with deeper spatiality-aware feature representations.The comparison identifies lost visual-context interaction in DR-Net and limited in-depth spatiality-aware representation in ViP.
- Transferability: Adding SCA-M to ViP improves performance by a considerable margin on all evaluation metrics, while combining SCA-M and IH-tree surpasses ViP itself.This supports combining feature-level contextual connections with ViP’s label-level message passing.
6 Comparisons on Visual Relationship Dataset (VRD)
On VRD, Zoom-Net is evaluated under matched prior-art settings using RPN proposals and triplet NMS. It outperforms prior methods on most metrics, while predicate-only k = 1 performance is merely comparable because competing methods use ground-truth subject and object labels.
- Evaluation settings: VRD comparisons use RPN-generated object proposals, triplet NMS, and Recall@50/100 with k = 70 plus k = 1 evaluation.Predicted boxes must exceed 0.5 IoU with ground truth.
- Results: 3.47% Rec@50 and 3.62% Rec@100 predicate-prediction gains are reported for k = 70.Relationship and phrase Rec@50 also increase by 1.25% and 6.46%, respectively.
- Results: Zoom-Net improves on most VRD evaluation metrics compared with state-of-the-art methods.The reported gains cover predicate, relationship, and phrase prediction tasks.
- Results: Predicate k = 1 performance is comparable with some prior methods because those methods use ground-truth subject and object labels.Zoom-Net predicts subject, predicate, and object jointly.
- Transferability: Replacing CAI’s visual representation with SCA-M increases predicate Rec@50 by 2.39%.The comparison is presented as evidence of stronger visual feature representation without relying on external linguistic data.
7 Conclusion
The paper concludes that Zoom-Net combines SCA-M feature learning with IH-trees for visual relationship recognition. It reports state-of-the-art performance on VG and VRD and identifies feature interactions as a direction for other vision applications.
- Framework: Zoom-Net uses SCA-M to learn spatiality-aware contextual feature representations for visual relationship recognition.SCA-M contains Contrastive ROI Pooling and Pyramid ROI Pooling Cells.
- Label handling: IH-trees model intra-class correlations to handle ambiguous and noisy relationship labels.The strategy targets annotation issues in the Visual Genome dataset.
- Conclusion: Zoom-Net achieves state-of-the-art performance on both VG and VRD datasets.The conclusion also reports superiority and transferability for each component.
- Future directions: The authors identify image retrieval and image caption generation as potential applications for feature interactions.This is presented as a direction for future exploration.
8 Appendix
The appendix documents label-frequency patterns, hierarchical label construction, loss-weight balancing, and supplementary dataset analyses. On VG, semantic clustering reduces object classes from 5,319 to 578, while VRD omits an intermediate hierarchy layer because its annotations are cleaner.
- Category frequencies: On VG, the most frequent object category is man and the most common predicate is on.Wordle font size represents category frequency.
- IH-tree construction: The IH-trees contain three levels, with H0 holding source dataset labels.VG object labels are clustered by semantic similarity in a higher layer.
- Dataset differences: The intermediate H1 layer is removed for VRD because its annotations are sufficiently clean to require less ambiguity and noise reduction.The layer is retained for addressing ambiguity and annotation noise in VG.
- Loss balancing: Loss weights α, β, and γ balance the three subject, object, and relationship branches during training.Because the branches interact through feed-forward and back-propagated updates, modest loss-weight differences do not dominate training.
Sensitivity to loss weights of multi-tasks.
The supplied appendix passages introduce additional qualitative results and scene graph generation, but they do not report sensitivity results for multi-task loss weights.
- Supplementary results: The appendix provides additional qualitative results on the Visual Genome dataset.The passage refers readers to the main paper for experimental settings and details.
- Scene graph generation: Zoom-Net is also applied to scene graph generation, which produces a directed graph capturing image objects and their relationships.The passage connects scene graph generation with visual question answering and image retrieval.
Scene Graph Generation
Zoom-Net generates scene graphs for visual relationship recognition, while qualitative examples expose both successful predictions and errors. The Visual Genome long-tail distribution creates zero-shot tuple relationships that are absent from training.
- Scene Graph Generation: Zoom-Net produces scene graphs containing predicted subjects, predicates, and objects for visual relationship recognition.Figure 8 marks incorrect predictions and links object-category colors in the graph to rectangle colors in the corresponding image.
- Qualitative Examples: Figure 8 visualizes two exemplar scene graphs generated by Zoom-Net, including incorrect subject, predicate, or object predictions.
- Zero-shot Relationships: Long-tail relationship labels make tuple combinations difficult to cover across training and testing, producing zero-shot relationships such as ⟨water-in-window⟩ and ⟨vase-on-head⟩.The individual object and predicate categories may appear in both splits even when their combination does not appear in training.
Zero-shot Relationship Recognition.
The qualitative evaluation compares Zoom-Net configurations, the effect of removing IH-trees, and performance against state-of-the-art methods. Results use Top-1 predictions for configuration and method comparisons, and Top-10 predictions for the IH-tree comparison.
- Module Configurations: Figure 9 compares different Zoom-Net module configurations using Top-1 prediction results.
- IH-tree Ablation: Figure 9 reports Top-10 triple relationship predictions for Zoom-Net with and without IH-trees.The comparison evaluates qualitative results after discarding the IH-trees.
- State-of-the-art Comparison: Figure 9 compares Zoom-Net with the state-of-the-art methods DR-Net and ViP using Top-1 prediction results.