PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Convolutional Scene Graph Generation

Hengyue Liu, Ning Yan, Masood S. Mortazavi, Bir Bhanu

arXiv:2103.16083v1cs.CV

TL;DR

Scene graph generation commonly relies on two-stage detectors, while high computational complexity and biased predictions remain concerns. FCSGG predicts objects and relationships together using object centers and relation affinity fields in a fully convolutional model. On Visual Genome, it reports the best scene-graph detection results among visual-only models and competitive overall SGG results.

  • Problem

    Common two-stage SGG pipelines are computationally complex, and long-tailed relationships and object-label-conditioned bias hinder recognition of less frequent and unseen relationships.

    Source: §1 · pp. 1–2

  • Method

    FCSGG uses a fully convolutional bottom-up model that represents objects as points and relationships as 2D relation affinity fields, predicting both from dense visual features.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    FCSGG achieves the best scene-graph detection results among visual-only models, although its SGG results do not outperform state-of-the-art approaches overall.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    The authors present FCSGG as a strong visual-only baseline with fast inference and generalizability to unseen relationships on Visual Genome.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

This paper presents a fully convolutional scene graph generation (FCSGG) model that detects objects and relations simultaneously. Most of the scene graph generation frameworks use a pre-trained two-stage object detector, like Faster R-CNN, and build scene graphs using bounding box features. Such pipeline usually has a large number of parameters and low inference speed. Unlike these approaches, FCSGG is a conceptually elegant and efficient bottom-up approach that encodes objects as bounding box center points, and relationships as 2D vector fields which are named as Relation Affinity Fields (RAFs). RAFs encode both semantic and spatial features, and explicitly represent the relationship between a pair of objects by the integral on a sub-region that points from subject to object. FCSGG only utilizes visual features and still generates strong results for scene graph generation. Comprehensive experiments on the Visual Genome dataset demonstrate the efficacy, efficiency, and generalizability of the proposed method. FCSGG achieves highly competitive results on recall and zero-shot recall with significantly reduced inference time.

1. Introduction

Scene graphs represent objects and their relationships explicitly, but common two-stage pipelines face long-tailed relationships, biased predictions, and high computational complexity. FCSGG instead predicts objects and relationships in one fully convolutional model, using object points and relation affinity fields, and reports strong visual-only results with near-real-time inference.

1. Introduction

  • Scene graphs explicitly represent object classes as nodes and relationships as edges, with triplets expressing subject, predicate, and object.The paper situates this representation as useful for downstream tasks including captioning, retrieval, visual question answering, and image generation.Source: §1 · p. 1
  • Existing SGG systems commonly detect objects first and then build a graph, while long-tailed relationships and object-label-conditioned bias hinder recognition of less frequent and unseen predicates.High computational complexity also impedes practicality, motivating per-pixel scene-graph prediction.Source: §1 · pp. 1–2
  • FCSGG models objects as points and relationships as 2D vector-field segments called relation affinity fields, predicting both as dense feature maps in one CNN.The representation is designed to preserve spatial information while encoding relationships between object pairs.Source: §1 · p. 2
  • The paper presents FCSGG as the first fully convolutional SGG model, designed to be more compact and computationally efficient than previous SGG models.The authors report near-real-time inference and benchmark model efficiency.Source: §1 · p. 2
  • The model reports better results than most visual-only SGG methods and competitive results against methods boosted by external knowledge.The introduction also describes the method as achieving strong zero-shot recall.Source: §1 · p. 2

2. Related Work

Prior SGG work refines contextual features, incorporates external knowledge, or explores other feature representations. FCSGG differs by grounding relationships directly in CNN features and predicting objects and relations in one forward pass without explicit iterative information exchange.

2. Related Work

  • Contextual-feature methods use recurrent networks or message passing, while spatial-feature approaches merge semantic levels or encode bounding-box geometry.The cited approaches include iterative message passing, visual-phrase-constrained message passing, bounding-box masks, and relative-position features.Source: §2 · p. 2
  • FCSGG grounds relationships semantically and spatially in CNN features, predicting objects and relationships in one forward pass without explicit iterative information exchange.The paper also distinguishes its visual-only relationship representation from latent-space relationship embeddings and reports near-real-time inference.Source: §2 · p. 2
  • External-knowledge approaches add linguistic or commonsense information, use object-statistics correlations, or apply causal interventions and long-tail mitigation.FCSGG relies only on visual features yet reports strong performance relative to these approaches.Source: §2 · p. 2

3. Object Detection as Keypoint Estimation

The section frames object detection as keypoint estimation: a CNN predicts dense object-center, offset, and size features, from which detections can be recovered. Center heatmaps use Gaussian-smoothed targets, while offsets correct for downsampling and size targets encode box dimensions.

3. Object Detection as Keypoint Estimation

  • The model builds on CenterNet, replacing anchor-based proposals with dense predictions for object centers, center offsets, and object sizes.A backbone produces feature maps that feed convolutional prediction heads; the section describes a default total stride of 4.Source: §3 · pp. 2–3
  • Object centers are represented by Gaussian-smoothed heatmaps at feature-map resolution, with Gaussian focal loss used to supervise the center heatmaps.The center coordinates are divided by the output stride when constructing the feature-level targets.Source: §3 · p. 3
  • Center offsets correct discretization error from downsampling, while object-size regression predicts box dimensions and recovers their actual size using the output stride.The regressors are class-agnostic; when two centers collide at one location, the smaller object is selected for regression.Source: §3 · p. 3

4. Relation Affinity Fields

FCSGG represents relationships as dense predicate-specific vector fields between object centers, enabling joint detection and relation training. At inference, it scores relationships by integrating those fields along paths between detected centers, while multi-scale assignment addresses center collisions and scale ambiguity.

4. Relation Affinity Fields

  • RAFs represent each predicate with vectors flowing from a subject center toward an object center, averaged where same-predicate relationships overlap.The model adds a branch that predicts these 2D vector fields as dense feature maps.Source: §4 · p. 4
  • Dense RAFs encode spatial and semantic features, support end-to-end joint object and relation detection training, and are learned with per-pixel weighted regression.Pixels on a relationship segment receive weight 1; nearby pixels receive distance-dependent weights, while other pixels have zero target vectors.Source: §4 · p. 4

4.1. Inference

  • For each detected-center pair, inference averages RAF projections along the connecting path to score relationships, combines scores with object-class scores, and ranks both directions.A negative integral indicates the opposite relationship direction; the method can also apply frequency bias.Source: §4.1 · p. 4

4.2. Multi-scale Prediction

  • Multi-scale feature levels assign objects by size to address center collisions and single-pixel regression ambiguity, choosing the smallest object if targets still coincide.The model uses four or five feature scales depending on input size.Source: §4.2 · p. 5
  • RAF training assigns relationships to scales by center distance; shared detection heads predict across scales, whose outputs are merged and ranked at inference.The paper reports that multi-scale ground-truth assignment and scale-aware training improve results over single-scale prediction by resolving the stated ambiguity.Source: §4.2 · p. 5

5. Experiments

Experiments on Visual Genome evaluate FCSGG’s scene-graph performance, generalization, design choices, and efficiency. The results show competitive visual-only performance, strong zero-shot results, and reduced model size and inference time.

5. Experiments

  • Experiments use VG-150, with 150 object and 50 predicate categories, and evaluate PredCls, SGCls, and SGDet without training separate models for each task.Source: §5 · p. 5

5.1. Implementation Details

  • The tested architectures include ResNet and HRNet backbones; ResNet50-4S-FPN×2 uses parallel FPNs for object detection and RAF features.Source: §5.1 · p. 6

5.2. Quantitative Results

  • FCSGG achieves the best SGDet results among visual-only models, with 13.8 / 17.9 gain on SGDet ng-R@50 / 100 over Pixels2Graphs using RPN.Source: §5.2 · p. 6, Table 1 · p. 6
  • On PredCls, ResNet50-4S-FPN×2 reaches 10.6 zsR@100 versus VCTree-TDE’s 17.6, with 36M versus 360.8M parameters and 40 ms versus 1.69 s inference time.Source: §5.2 · p. 7, Table 3 · p. 7
  • Frequency-biased HRNetW48-5S-FPN×2-f improves R@20 by 2.6 over HRNetW48-5S-FPN×2 using training-set statistics.Source: §5.2 · p. 7

5.3.1 RAF regression Loss

  • Using L1 with β = 10 performs better; supervising only positive RAF locations gives comparable mean recall and detects more semantic and rare relationships, while negative loss favors ON and HAS.Source: Table 4 · p. 7, §5.3.1 · p. 8

5.3.2 Architecture Choices

5.3.3 Model Size and Speed

  • FCSGG models have considerably fewer parameters and lower inference time than prior work; the backbone dominates computation despite up to 49,500 candidate relationships per image.Source: §5.3.3 · p. 8, Table 6 · p. 8

6. Conclusions

FCSGG is a fast, fully convolutional scene-graph framework that uses relation affinity fields to represent visual relationships. On Visual Genome, it reports competitive object-detection and scene-graph results with visual features alone, including generalization to unseen relationships.

6. Conclusions

  • FCSGG is a simple, effective scene-graph framework with fast inference speed.Source: §6 · p. 8
  • Relation affinity fields provide a representation for visual relationships that generalizes to unseen relationships.Source: §6 · p. 8
  • Using visual features alone, FCSGG achieves competitive object-detection and SGG results on the VG dataset.Source: §6 · p. 8

Appendix

  • The supplementary document covers training details, network inference, architectures, and qualitative studies.Source: Appendix · p. 12

Appendix A. Training

Training turns object centers into Gaussian heatmaps and supervises them with Gaussian focal loss, while inference extracts centers and reconstructs boxes from predicted offsets and sizes. Multi-scale detections are merged through per-class NMS.

Appendix A. Training

  • The center heatmap uses Gaussian masks whose radii are chosen so centers within each ellipse yield boxes with at least 0.5 IoU against ground truth.The Gaussian standard deviations are computed from the desired x- and y-axis radii.Source: Appendix A · p. 12
  • Gaussian focal loss supervises predicted center heatmaps with α = 2 and γ = 4 for weight balancing.The loss is defined pixel-wise for the predicted center heatmap.Source: Appendix A · p. 12

Appendix B. Inference

  • At inference, FCSGG filters center heatmaps with sigmoid and 3 × 3 max pooling, keeps the top 100 peaks per class, and uses predicted offsets and sizes to recover boxes.For multi-scale prediction, per-class NMS is applied before retaining the top 100 boxes across scales.Source: Appendix B · p. 12

B.1. Path Integral

The path integral is computed on scale-specific RAFs for valid object pairs, with sampled points used to implement the integral efficiently.

B.1. Path Integral

  • For multi-scale RAFs, valid object pairs are selected from the kept detections; pairs within distance [0, 64] use the 5-scale RAF F3.The top 100 relationships from each scale are gathered, then the top 100 across scales are retained for evaluation.Source: §B.1 · p. 12
  • The integral is implemented with matrix multiplication using a shared number of sampled path points set by the longest integral among predicted object pairs.Each pair receives mmax sampled points regardless of its distance.Source: §B.1 · p. 12

B.2. Performance Upper Bound

With ground-truth objects and RAFs, evaluation tests whether path integration can recover scene graphs, reaching high but not complete recall.

B.2. Performance Upper Bound

  • 91.13 R@20 and 86.85 mR@20 are achieved using ground-truth objects and RAFs to evaluate scene-graph recovery on the test set.The paper attributes the non-100% recall ceiling to annotations containing multiple edges between nodes.Source: §B.2 · p. 12
  • Multiple edges between nodes in some ground-truth annotations make 100% recall impossible in this evaluation.Source: §B.2 · p. 12

Appendix C. Detailed Architectures

The appendix documents FCSGG's implementation and model configurations, including backbone choices and their feature architectures.

Appendix C. Detailed Architectures

  • The implementation is based on Detectron2 and Tang et al., and the model-zoo table lists architectures and prediction-head channel counts.The table also reports parameter counts, with the backbone having the most parameters and the heads relatively few.Source: Appendix C · pp. 12–13, Table 7 · p. 13

C.1. Backbone

  • ResNet progressively downsamples feature maps while increasing channels, whereas HRNet maintains higher-resolution branches and repeatedly fuses their features.The paper uses the original backbone architectures and hyper-parameters.Source: §C.1 · p. 13

C.2. Neck

The neck constructs feature representations at multiple scales or fuses HRNet branches for prediction. Four heads separately predict object centers, offsets, sizes, and relation affinity fields.

C.2. Neck

  • For ResNet, FPN exchanges information across scales to build P2–P5 feature maps, while the modified BiFPN adds more cross-scale connections.The FPN upsamples higher-level features and sums them with lower-level features.Source: §C.2 · p. 13
  • For HRNet, the single-scale neck upsamples and concatenates C2–C5 features, while HRNetV2p adds pooling and convolutions to construct multiple scales.The single-scale feature-fusion module has no trainable parameters and feeds the fused features directly to the heads.Source: §C.2 · pp. 13–14, Figure 6 · p. 14

C.3. Heads

  • Four heads predict center heatmaps, center offsets, object size, and relation affinity fields, using four convolutional blocks and task-specific output channels.The first three are object-detection heads; the last is the relation-detection head.Source: §C.3 · p. 14

Appendix D. Qualitative Results

Qualitative results show object localization and relationship predictions on Visual Genome images, alongside challenges from ambiguous object and predicate labels. RAF visualizations also illustrate responses on wild images, including a WEARING response strongest for a person wearing a jacket.

Appendix D. Qualitative Results

  • RAF visualizations encode vector orientation by hue and vector length by saturation, showing boxes above 0.2 and relationships above 0.1 scores.Figure 7 presents the RAF color coding.Source: Appendix D · p. 14, Figure 7 · p. 14
  • On Visual Genome, subtle object-class distinctions can produce misclassifications, including person labels that make associated relationships false detections.The examples include jacket/coat and man/woman confusions.Source: Appendix D · p. 14
  • Predicate ambiguity includes semantically similar or vague labels, yet FCSGG predicts similar responses for related predicates and shows relationship generalization.The examples include WEARING/WEARS responses and unannotated relationships predicted in Figure 9c.Source: Appendix D · pp. 14–15
  • Multiple predicates can describe one object pair, and RAF visualizations show reciprocal predicates such as ABOVE/UNDER with similar magnitudes and opposite directions.The passage also notes a similar response pattern for OF/HAS.Source: Appendix D · p. 15
  • For three wild jacket images, the unnormalized WEARING RAF norm is strongest for the man wearing a jacket, while WEARING responses also appear without a person.The reported norms are 0.79, 4.02, and 0.67 for the product, worn-jacket, and jacket-on-bed images, respectively.Source: Appendix D · p. 15, Figure 8 · p. 15
  • FCSGG localizes objects strongly, including a pole absent from ground truth and more accurate detections than ground truth in another example.Figure 9 shows input images, ground-truth and predicted objects, scene graphs, and selected predicted RAFs.Source: Appendix D · pp. 14, 17, Figure 9 · p. 18

Related papers

Loading 2103.16083v1…