Source-linked AI summary
Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
Yikang Li, Wanli Ouyang, Bolei Zhou, Jianping Shi, Chao Zhang, Xiaogang Wang
TL;DR
Scene graph generation is slowed by quadratic pairwise phrase representations and can become intractable as relationship combinations grow. F-Net factorizes overlapping relation regions into shared subgraphs, preserves spatial information with 2-D features, and uses SMP and SRI for refinement and recognition. Experiments on Visual Relationship Detection and Visual Genome report faster inference with better results than prior methods.
Problem
Pairwise phrase representations create quadratic intermediate computation, while integrated visual-phrase classification becomes intractable as object–predicate combinations grow.
Method
F-Net clusters object pairs with similar interacting regions into shared subgraphs, uses 2-D subgraph maps, and applies SMP and SRI for relation inference.
Results
F-Net is significantly faster than previous methods with better results on Visual Relationship Detection and Visual Genome.
Takeaways & Limitations
Shared subgraph representations reduce intermediate representations while retaining spatial information for scene graph relationship recognition.
Takeaways & Limitations
Integrated visual-phrase methods become intractable when combinations of object categories and relationship predicates become very large.
Abstract
from arXiv · showhide
Generating scene graph to describe all the relations inside an image gains increasing interests these years. However, most of the previous methods use complicated structures with slow inference speed or rely on the external data, which limits the usage of the model in real-life scenarios. To improve the efficiency of scene graph generation, we propose a subgraph-based connection graph to concisely represent the scene graph during the inference. A bottom-up clustering method is first used to factorize the entire scene graph into subgraphs, where each subgraph contains several objects and a subset of their relationships. By replacing the numerous relationship representations of the scene graph with fewer subgraph and object features, the computation in the intermediate stage is significantly reduced. In addition, spatial information is maintained by the subgraph features, which is leveraged by our proposed Spatial-weighted Message Passing~(SMP) structure and Spatial-sensitive Relation Inference~(SRI) module to facilitate the relationship recognition. On the recent Visual Relationship Detection and Visual Genome datasets, our method outperforms the state-of-the-art method in both accuracy and speed.
1 Introduction
Scene graph generation is useful for structured scene understanding and downstream applications, but pairwise phrase features create quadratic computation; F-Net addresses this with shared subgraph representations and spatially informed refinement.
- Scene graphs abstract objects and pair-wise relationships, supporting higher-level scene understanding and applications such as image retrieval and visual question answering.
- Pairwise phrase features become intractable as object counts grow because candidate combinations increase quadratically.Using fewer objects or filtering pairs can sacrifice the model’s performance upper bound.
- F-Net clusters object pairs referring to highly overlapped regions into subgraphs that share phrase representations during feature refinement.This replaces repeated phrase processing with a shared representation refined using connected-object messages.
- Shared subgraph features significantly reduce intermediate phrase features and speed model training and inference.
- F-Net uses 2-D subgraph feature maps, Spatial-weighted Message Passing, and Spatial-sensitive Relation Inference to preserve and exploit spatial information.
2 Related Work
Prior visual relationship methods range from integrated visual-phrase classification to separate predicate and object-category modeling, with closely related work also jointly refining scene-graph representations.
- Early visual relationship studies focused on specific spatial relations or object interactions and often used hand-crafted features.
- Integrated visual-phrase methods treat each object-category and predicate combination as a distinct class, becoming intractable when combinations grow large.
- Recent methods increasingly model relationship predicates and object categories separately rather than as one integrated class.
- Generic visual relationship detection detects objects first and recognizes predicates between object pairs, using object-category word embeddings as language priors.
- The most related prior methods jointly detect objects and recognize relationships through iterative feature refinement or by adding region captioning as a higher-semantic-level task.
3 Framework of the Factorizable Network
F-Net builds a factorized connection graph by proposing objects, enumerating directed object pairs, clustering similar relation regions, and jointly refining object and subgraph features before prediction.
- F-Net begins with Region Proposal Network object proposals and groups proposals into pairs connected by two directed edges per object pair.
- Similar phrase-region edges are clustered into subgraphs, producing a more concise connection graph that shares representations across candidate relations.
- ROI pooling represents objects as feature vectors and subgraphs as 2-D feature maps, after which messages refine both feature types.
- Object categories are predicted from object features, while predicates are inferred by fusing subgraph features with subject–object feature pairs.
- The fully connected graph contains N(N −1) candidate relations for N object proposals, creating a speed–recall trade-off as proposal counts increase.
3.3 Factorized Connection Graph Generation
F-Net factorizes overlapping relation regions into shared subgraph representations, retains their spatial structure with 2-D maps, and refines them through spatially weighted interactions.
- Factorized Connection Graph Generation: Candidate relations with highly overlapped union regions are merged using confidence scores, bounding-box locations, and non-maximum suppression.The highest-scoring similar box is retained as the representative for a subgraph.
- Factorized Connection Graph Generation: Each subgraph contains several objects, each object belongs to several subgraphs, and each candidate relation maps to one subgraph and two objects.
- Factorized Connection Graph Generation: Subgraph features use 2-D feature maps rather than vectors to maintain spatial information within shared regions.Because the representation is shared across predicate inferences, the map can learn a more general regional representation.
3.5 Feature Refining with Spatial-weighted Message Passing
The SMP structure passes messages between object feature vectors and subgraph feature maps, leveraging their complementary semantic levels. It uses different pooling strategies because objects need general subgraph information rather than spatial detail.
- 3.5 Feature Refining with Spatial-weighted Message Passing: SMP exchanges messages between object vectors and subgraph feature maps to leverage their complementary semantic information.Objects concentrate on details, while subgraphs focus on interactions.
- Pass Message From Subgraphs to Objects: Subgraph-to-object messages are converted from 2-D feature maps into feature vectors using average pooling.Objects require general information about subgraph regions instead of their spatial information.
Pass Message From Subgraphs to Objects
Messages from connected subgraphs are pooled and attention-weighted before refining each target object feature. This produces object representations informed by multiple subgraph regions.
- Pass Message From Subgraphs to Objects: Each connected subgraph feature map is average-pooled into a feature vector before message aggregation.The pooled vector is denoted sk.
- Pass Message From Subgraphs to Objects: Attention weights aggregate subgraph features for each target object when the object connects to multiple subgraphs.The aggregation is represented by pi(Sk) over the connected subgraphs Si.
- Pass Message From Subgraphs to Objects: The attention transformation maps subgraph vectors into the target object's feature domain and applies a ReLU layer.FC(att s) transforms sk to the domain of oi.
- Pass Message From Subgraphs to Objects: The aggregated message features are passed through FC(s→o) to refine the target object feature.The resulting representation is denoted oî.
Pass Message From Objects to Subgraphs
Objects send spatially weighted messages to subgraph feature maps. Location-dependent probabilities determine how each connected object contributes across the subgraph's spatial dimensions.
- Pass Message From Objects to Subgraphs: Object features are passed to a subgraph's 2-D feature map with weights that depend on object locations.The process aggregates several object feature vectors into a spatial feature map.
- Pass Message From Objects to Subgraphs: For subgraph k, the aggregated message at location (x, y) sums each connected object feature oi weighted by its probability map Pk(oi)(x, y).Ok is the set of objects contained in subgraph k.
- Pass Message From Objects to Subgraphs: The object-to-subgraph transformation maps each object feature into the target subgraph domain before spatial aggregation.The transformed messages are combined with the subgraph feature at each location.
- Pass Message From Objects to Subgraphs: Object-message probabilities are normalized across objects at each location, while different spatial locations remain unconstrained.Thus, different objects can refine different parts of the subgraph feature.
- Pass Message From Objects to Subgraphs: The aggregated object messages form a feature map that is used to refine the subgraph features through a convolution layer.The convolution transforms merged object messages into the target subgraph domain.
- Pass Message From Objects to Subgraphs: Multiple SMP modules can be stacked to enhance the model's representation ability.The passage presents stacking as an additional design option after spatially weighted message passing.
3.6 Spatial-sensitive Relation Inference
Spatial-sensitive relation inference uses refined subject, object, and subgraph features to extract relationship cues from subgraph feature maps. A bottleneck convolution reduces predicate-classification parameters, while spatial information remains embedded in the maps.
- 3.6 Spatial-sensitive Relation Inference: Refined object features predict object categories, while subject, object, and subgraph features jointly predict relationships.Subgraph features can correspond to several object pairs, so the pair features are combined with their corresponding subgraph feature.
- 3.6 Spatial-sensitive Relation Inference: Subject and object features act as convolution kernels that extract relationship cues from the corresponding subgraph feature map.This uses the spatial correspondence between objects and regions of the subgraph feature.
- 3.6 Spatial-sensitive Relation Inference: Group convolution is adopted for kernel computation and reformulated as an element-wise product when group numbers equal the channel count.The design avoids learning a large conventional convolution kernel directly.
- 3.6 Spatial-sensitive Relation Inference: Predicate recognition concatenates the convolution result with the subgraph feature and applies a fully-connected layer.FC(p) denotes the predicate-recognition layer.
- 3.6 Spatial-sensitive Relation Inference: A 1 × 1 bottleneck convolution reduces channels before predicate classification, lowering the parameter size from the direct formulation.The bottleneck uses channel size C′.
- 3.6 Spatial-sensitive Relation Inference: With C′ = C/2, the bottleneck design almost halves the number of parameters because the bottleneck convolution is much smaller than FC(p).This is the paper's stated parameter reduction condition.
- 3.6 Spatial-sensitive Relation Inference: Unlike prior coordinate- or mask-based spatial features, this method embeds spatial information directly in subgraph feature maps.FC(p) can learn how to use location-specific information from training data.
4 Experiments
The experiments evaluate F-Net through ablations and comparisons on VRD and cleansed Visual Genome datasets, measuring accuracy and testing speed across phrase detection and scene graph generation.
- Datasets: F-Net is evaluated on VRD and two cleansed Visual Genome versions for fair comparison with existing methods.The datasets address VRD benchmarking and noise in raw Visual Genome labels.
- Tasks: Evaluation covers Visual Phrase Detection and Scene Graph Generation, with both tasks recognizing subject-predicate-object triplets.Scene graph generation additionally localizes both subject and object regions.
- Metrics: Rec@50 and Rec@100 measure how many labeled relationships appear among the top K predictions because relationship annotations are incomplete.Recall is used instead of mAP to avoid penalizing positive but unlabeled relations.
- Metrics: Testing speed is reported as a critical metric alongside accuracy because scene graph generation is approaching practical applications and products.Unless specified otherwise, speed is measured on a Titan-X GPU.
- Ablation studies: The ablation table reports whether subgraph clustering, 2-D feature maps, SMP, and SRI are used, alongside proposal counts and seconds per image.These settings isolate the contributions of the proposed representation and modules.
- Ablation studies: Subgraph-based models 1–6 are all faster than phrase-feature baseline model 0, while model 2 exceeds model 0 in both speed and accuracy.The comparison attributes the speed advantage to fewer intermediate features, while additional object proposals compensate for approximation effects.
2-D feature map
F-Net uses 2-D subgraph feature maps to preserve spatial structure, with SMP and SRI refining representations for relation inference while exposing accuracy–speed trade-offs.
- 2-D feature map: 2-D feature maps preserve spatial information within subgraph regions instead of reducing subgraph features to vectors.Models using these maps replace fully connected transformation with convolutional processing.
- 2-D feature map: Compared with model 2, model 3 performs better using 2-D representations and average-pooled subgraph features for relationship prediction.The spatial structure can act as a regularization, while the convolutional design uses fewer learnable parameters.
- SMP: SMP increases SGGen Recall by 2.02%–2.37% when comparing models 3 and 4.SMP refines different subgraph parts with different objects and updates object features using interaction information.
- SMP: Stacking more SMP modules further improves performance but deteriorates testing speed, especially with feature-map representations.The speed cost is associated with introducing more complicated message paths.
- SRI: SRI assigns location-specific weights to points in the 2-D map to model hidden spatial connections for relationship prediction.The module is designed to exploit spatial structure without relying on handcrafted spatial features.
- Comparison: F-Net is compared with existing methods on PhrDet and SGGen using VRD and two cleansed Visual Genome benchmarks, with speed measured per image.The comparison includes both accuracy-oriented tasks and testing-time efficiency.
- Comparison: The subgraph pipeline reduces relationship representations, allowing more object proposals and supporting better speed and accuracy than comparison methods.Prior methods represent each relationship with an individual phrase feature, causing relationship counts to grow quadratically with proposals.
5 Conclusion
The paper concludes that F-Net improves scene graph generation efficiency by replacing quadratic relationship representations with concise subgraph features while preserving spatial information.
- Conclusion: F-Net reduces intermediate representations through a concise subgraph-based scene graph representation.The approach targets the quadratic number of possible relationships during inference.
- Conclusion: 2-D feature maps preserve spatial information within subgraph regions for relation recognition.SMP and SRI exploit the inherent spatial structure of these feature maps.
- Conclusion: Experiments report that F-Net is significantly faster than previous methods with better results.The conclusion summarizes improvements in both testing efficiency and scene graph generation outcomes.