PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoptic Scene Graph Generation

Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, Ziwei Liu

arXiv:2207.11247v1cs.CVcs.AIcs.CLcs.LGcs.MM

TL;DR

Bounding-box-based SGG can provide coarse or incomplete grounding and overly redundant dataset information, limiting scene graph representations. The paper introduces PSG, builds a 49k-image benchmark with panoptic segmentations, and evaluates two-stage and one-stage baselines. One-stage models achieve competitive results, while PSGFormer improves mean recall over the best two-stage model under a limited training schedule.

  • Problem

    Bounding-box-based SGG provides coarse localization, omits background regions, and includes redundant groundings or predicates in existing datasets.

  • Method

    The paper introduces PSG, constructs a COCO–Visual Genome dataset with panoptic segmentations, and benchmarks two-stage and one-stage models including PSGTR and PSGFormer.

  • Results

    One-stage models achieve competitive results, while PSGFormer outperforms the best two-stage model by 4.8% on mR@20 and 8.5% on mR@100.

  • Takeaways & Limitations

    Panoptic scene graphs provide a benchmark for more precise and comprehensive scene representations, including relations involving background stuff.

  • Takeaways & Limitations

    PSG models should rely heavily on visual clues and still need to predict more meaningful and diverse relations, including rare relations.

Abstract

from arXiv · show

Existing research addresses scene graph generation (SGG) -- a critical technology for scene understanding in images -- from a detection perspective, i.e., objects are detected using bounding boxes followed by prediction of their pairwise relationships. We argue that such a paradigm causes several problems that impede the progress of the field. For instance, bounding box-based labels in current datasets usually contain redundant classes like hairs, and leave out background information that is crucial to the understanding of context. In this work, we introduce panoptic scene graph generation (PSG), a new problem task that requires the model to generate a more comprehensive scene graph representation based on panoptic segmentations rather than rigid bounding boxes. A high-quality PSG dataset, which contains 49k well-annotated overlapping images from COCO and Visual Genome, is created for the community to keep track of its progress. For benchmarking, we build four two-stage baselines, which are modified from classic methods in SGG, and two one-stage baselines called PSGTR and PSGFormer, which are based on the efficient Transformer-based detector, i.e., DETR. While PSGTR uses a set of queries to directly learn triplets, PSGFormer separately models the objects and relations in the form of queries from two Transformer decoders, followed by a prompting-like relation-object matching mechanism. In the end, we share insights on open challenges and future directions.

1 Introduction

The paper identifies limitations in bounding-box-based scene graph generation and introduces panoptic scene graph generation to represent objects, background, and relationships more precisely. It also presents a 49k-image dataset and baselines for benchmarking the new task.

  • Scene graph generation supports image retrieval, visual reasoning, VQA, captioning, image generation, and robotics by representing objects and their pairwise relationships.
  • Bounding-box-based SGG provides coarse, noisy localization, omits important background regions, and includes redundant information such as person-has-hair.
  • PSG replaces rigid bounding boxes with panoptic segmentations to provide clearer grounding and more comprehensive scene representations.
  • The PSG dataset contains 49k well-annotated images from COCO and Visual Genome, using 133 object classes and 56 predicate classes.
  • The benchmark includes four two-stage baselines and one-stage Transformer-based models, including PSGTR and PSGFormer.
  • One-stage models achieve competitive results on the PSG dataset despite using a simplified training paradigm.

2 Related Work

Prior SGG research largely relies on bounding-box grounding and datasets with problematic class and predicate definitions. PSG addresses these issues through panoptic grounding and a more controlled predicate vocabulary.

  • Classic SGG methods generally use two-stage pipelines that detect objects and then estimate pairwise predicates.
  • The PSG benchmark compares dataset statistics using measures including predicates per image, duplicate-free groundings, and grounding supervision type.
  • Visual Genome contains 33,877 object classes and 40,480 predicate classes, motivating later datasets with more constrained vocabularies.
  • PSG uses a 56-class predicate dictionary designed to reduce overlap while retaining semantic coverage.
  • Bounding-box grounding causes coarse localization, incomplete background coverage, trivial relations, and duplicate groundings in SGG datasets.
  • Panoptic segmentation unifies semantic and instance segmentation to support comprehensive scene understanding.

3 Problem and Dataset

PSG formulates scene graph generation over non-overlapping panoptic masks, object classes, and relations rather than bounding boxes. Its dataset fuses COCO and Visual Genome annotations, and evaluation centers on scene graph generation from scratch.

  • Recap: Scene Graph Generation: Classic SGG models the scene graph as bounding boxes, object labels, and relations conditioned on the input image.
  • Panoptic Scene Graph Generation: PSG grounds both foreground objects and background stuff with panoptic segmentation masks, while predicting their class labels and relations.
  • Panoptic Scene Graph Generation: PSG instead models the graph distribution over masks, object labels, and relations conditioned on the image.
  • PSG Dataset: The dataset begins with 48,749 images shared by COCO and Visual Genome and automatically matches COCO segmentations with Visual Genome boxes.
  • Evaluation and Metrics: Scene graph generation is the main PSG evaluation task, while predicate classification is applicable only to two-stage models with given object labels and localization.
  • Evaluation and Metrics: PSG triplet recall requires correct subject and object masks with mask IoU above 0.5 and correct labels in every subject-verb-object position.

4 PSG Baselines

The PSG baselines adapt classic two-stage SGG pipelines and DETR-style one-stage architectures to predict panoptic scene graphs. PSGTR directly represents triplets with queries, while PSGFormer separately models objects and relations before matching them into triplets.

  • Two-Stage PSG Baselines: Two-stage PSG baselines first extract panoptic objects, masks, classes, and relation features, then apply classic SGG relation modules for final graph prediction.Panoptic FPN supplies initial representations; existing relation models produce relation triplets.
  • Two-Stage PSG Baselines: The two-stage formulation decomposes graph prediction into panoptic masks, object labels conditioned on masks, and relations conditioned on objects and masks.This decomposition is expressed as Pr(G | I) = Pr(M | I) · Pr(O | M, I) · Pr(R | O, M, I).
  • PSGTR: PSGTR uses CNN features, positional encoding, and triplet queries processed by a Transformer encoder-decoder to predict subject, predicate, object, and panoptic masks jointly.Separate FFNs predict triplet classes, while panoptic heads produce subject and object segmentations.
  • PSGTR: PSGTR extends DETR matching to triplets by matching query outputs against ground-truth triplets using class and segment costs.The matching considers subject, relation, object, and their localizations before optimizing the total loss.
  • PSGFormer: PSGFormer separately models object and relation queries with distinct Transformer decoders, then uses prompting-like selectors to compose subject-predicate-object triplets.Subject and object selectors use relation-query associations to choose object-query candidates in different roles.
  • PSGFormer: PSGFormer selects subjects and objects for each relation query, producing matched triplets T = {(S_i, R_i, O_i)}.The selectors use cosine similarity while applying separate subject and object representations.

5 Experiments

The experiments compare two-stage and one-stage PSG baselines using recall, mean recall, and panoptic segmentation quality. One-stage models show competitive or strong relation prediction, while segmentation quality and training duration remain important factors.

  • Main Results: Table 2 compares adapted two-stage SGG models and PSGFormer using recall and mean recall under 12-epoch training by default.Predicate classification is unavailable for one-stage models, and 60-epoch results are marked separately.
  • Two-Stage Baselines: Ground-truth segmentations substantially improve two-stage predicate prediction; IMP reaches over 30% R@20 in PredCls but falls from 32% to 17% on SGDet.The comparison indicates that two-stage PSG performance depends strongly on first-stage segmentation quality.
  • Two-Stage Baselines: GPSNet does not exceed MOTIFS and VCTree in PSG, suggesting its direction-modeling advantage transfers poorly when PSG removes trivial directional predicates.Examples of removed trivial predicates include hair-of-man and man-has-head.
  • Segmentation Quality: One-stage models produce more reasonable and diverse triplets but do not achieve good panoptic segmentation quality in the reported visualization.Figure 6 reports panoptic quality alongside predicted triplets and segmentation masks.
  • One-Stage Models: PSGFormer exceeds VCTree by 4.8% on mR@20 and 8.5% on mR@100, although it remains behind two-stage methods on R@20/50/100.Its higher mean recall is associated with unusual but accurate relation predictions.
  • One-Stage Models: PSGFormer’s separate object and relation decoders let relation queries independently capture predicate meaning, supporting unusual predictions.The reported examples include person-going down-snow, person-driving-bus, and person-wearing-skis.
  • One-Stage Models: PSGTR achieves state-of-the-art results after 60 epochs despite one-digit recall scores at 12 epochs, with triplet queries jointly predicting each triplet.Its cross-attention mechanisms expose triplet queries to information from the entire image.

6 Challenges and Outlook

The paper identifies visual grounding and relation diversity as open challenges for PSG. Models should use priors cautiously while relying heavily on visual evidence and capturing meaningful rare relations.

  • Challenges: PSG predictions should rely heavily on visual clues even when prior knowledge improves performance.The paper uses distinguishing walking from standing as an example of the visual evidence required for person-walking-on-pavement.
  • Challenges: Future PSG models should predict meaningful and diverse relations, including rare relations such as feeding and kissing, rather than only common or positional relations.The paper frames this as a direction for knowledge-aided models in multimodal settings.
  • Relation and Segmentation: PSGTR can achieve good PSG performance while obtaining miserable panoptic quality because triplet queries independently produce object groundings.Independent grounding can cause one object to be referred to and segmented by several triplets.

A.1 More comparisons between VG and PSG

The appendix compares VG-150 and PSG examples to show that PSG captures more representative, comprehensive, and accurate scene information. Its predicate design favors appropriate granularity over excessive specificity.

  • Dataset Comparison: VG-150 omits the key relation woman flying kite, whereas PSG captures it with the more general predicate playing.PSG defines predicates as representative with proper granularity, not too specific.
  • Dataset Comparison: PSG avoids ambiguous relations such as at and gathers more comprehensive and accurate triplets than VG-150.The comparison attributes the predicate difference to PSG’s representative-granularity rule.
  • Object Grounding: VG-150 can provide inaccurate object grounding, such as a beach mask covering only half the actual beach, whereas PSG improves grounding quality.Incorrect grounding can reduce triplet recall because successful matching requires high overlap with the ground-truth grounding.

A.2 PSG Dataset Statistics

The PSG dataset contains 48,749 annotated images with 56 predicate classes and COCO’s 80 thing and 53 stuff classes. Each image averages 11.0 instances and 5.6 relations, with thing-stuff relations most common.

  • 48,749 images comprise the PSG dataset, with 56 predicate classes, 80 thing classes, and 53 stuff classes.
  • Each image contains an average of 11.0 instances and 5.6 relations.
  • Thing-stuff relations average 2.5 per image, representing 45% of relations.
  • Thing-thing relations average 1.9 per image, representing 34% of relations.
  • Stuff-stuff relations average 1.2 per image, representing 21% of relations.

A.3 PSG Dataset Construction Details

The PSG dataset merges COCO panoptic annotations with Visual Genome relationships through category and instance matching, followed by human cleaning and supplementation. Its 56-predicate dictionary prioritizes representative, practical, and broad visual relations.

  • Dataset merging: PSG construction exploits 48,749 COCO–Visual Genome overlap images containing panoptic segmentations and scene-graph annotations.
  • Dataset merging: Dataset merging matches COCO and Visual Genome object categories and instances before transferring Visual Genome relationship annotations.
  • Annotation process: Transferred annotations undergo final human cleaning, filtering incorrect triplets and adding object-object, object-background, and background-background relations.
  • Dataset merging: Category matching uses fastText embeddings and cosine similarity, while instance matching greedily prioritizes bounding-box IoU among candidate pairs.
  • Predicate dictionary: The predicate dictionary deduplicates overlapping relations, removes many positional predicates, and selects 56 classes for representative, practical, and broad coverage.
  • Predicate dictionary: The dictionary includes positional relations, object-object relations, common actions, human actions, traffic actions, and sports actions.
  • Predicate dictionary: Detailed predicate definitions and image examples support consistent annotator performance.

B Implementation Details

Implementation uses a unified MMDetection codebase and combines dataset statistics with training procedures for panoptic and scene-graph models. PSGTR and PSGFormer follow DETR-style Transformer implementations with query-based prediction.

  • Reproducibility: All experiments use a single unified MMDetection codebase to facilitate reproducibility.
  • Dataset statistics: The dataset statistics include predicate-class proportions and distributions of relation counts and relation density per image.
  • Two-stage implementation: The Panoptic FPN base model is fine-tuned on PSG panoptic annotations before its weights are frozen for scene-graph-head training.
  • One-stage implementation: PSGTR extends DETR with new heads and a triplet Hungarian matcher, using 100 queries to predict possible relations.
  • One-stage implementation: PSGTR follows DETR training with AdamW optimization, COCO-pretrained initialization, and cropping and resizing augmentation.
  • One-stage implementation: PSGFormer uses separate object and relation queries, shares PSGTR’s training settings, and adds an auxiliary panoptic task.

C Visualization of PSGTR Result Triplet-by-Triplet

PSGTR’s triplet-by-triplet visualizations appear good even though its merged panoptic visualization is chaotic. Independent subject and object masks across triplets make re-identification and merging difficult.

  • PSGTR’s triplet-by-triplet results look good despite chaotic merged panoptic visualizations.
  • The visualization issue reflects PSGTR’s direct triplet-query design rather than an inability to segment objects well within individual triplets.
  • Independent subject and object masks across triplets make re-identification non-trivial and can split one object during pixel-wise argmax merging.
Loading 2207.11247v1…