Learning Canonical Representations for Scene Graph to Image Generation

1 Introduction

• Different scene graphs can encode the same physical configuration, yet existing models often fail to produce the same layout and image for equivalent graphs.For example, (A, right, B) entails (B, left, A), but human graphs typically include only one relation; the paper reports that only a small fraction of equivalent relations are annotated in practice.Source: §1 · p. 2
• The method replaces each input with a learned canonical scene graph, so logically equivalent graphs share a representation before layout generation.It learns this process from data without prior knowledge of relation properties and uses few additional parameters.Source: §1 · p. 2
• The model improves generation on Visual Genome, COCO, and CLEVR over state-of-the-art baselines, with improved robustness to graph size and noise.The contribution summary reports these benchmark gains alongside stronger invariance and generalization to semantically equivalent graphs.Source: §1 · p. 3

2 Related Work

• Prior scene-graph-to-image methods degrade on complex graphs, while this work addresses representation invariance through canonicalization rather than stronger coarse-grid supervision.Scene graphs encode attributes and relations, whereas layouts provide spatial information; the paper positions its invariance approach apart from permutation invariance.Source: §2 · p. 3
• The proposed pipeline canonicalizes the input graph, applies a GCN to weighted completed relations for bounding boxes, then uses those outputs to form a layout.Source: Figure 2 · p. 4

3 Scene Graph Canonicalization

• Canonicalization replaces logically equivalent scene graphs with a shared closure, so graph differences that do not change implied relations do not affect the representation.The closure contains relations that are true in any graph containing the input edges and satisfying the formula set.Source: §3 · p. 5

3.1 Calculating Scene Graph Canonicalization

• Under restricted transitive and converse formulas, SGC adds converse edges and computes transitive closure to produce the graph closure.General first-order closure is hard; the described transitive closure uses Floyd–Warshall, and symmetric relations are omitted because they are absent from the considered datasets.Source: §3.1 · p. 5

3.2 Calculating Weighted Scene Graph Canonicalization

• When relation properties are unknown, weighted canonicalization assigns learned probabilities to transitivity and converse relations and uses them to weight completed edges.The probabilities are parameterized and learned end to end.Source: §3.2 · p. 6
• WSGC-E weights converse edges by their converse probability and transitive completions by the maximum path weight times the transitivity probability, while WSGC-S samples converse edges at weight 1 before transitive completion.WSGC-S computes transitive closure on a sparser graph and assigns new transitive edges weight ptrans.Source: §3.2 · pp. 6–7, Figure 3 · p. 7

4 Scene Graph to Image using Canonicalization

• The pipeline first maps the completed weighted graph to a layout and then maps that layout to an image using variants of prior scene-graph-to-image components.The weighted graph is the input to the layout stage.Source: §4 · p. 7
• A graph convolutional network maps the weighted scene graph to object bounding boxes, using canonical-graph weights in its node aggregation.The aggregation is a weighted average.Source: §4 · pp. 7–8
• For CLEVR, AttSPADE extends layout-to-image generation to accept object attributes such as color, shape, and material.For non-CLEVR datasets, the experiments use a pretrained LostGAN model.Source: §4 · p. 8

5 Losses and Training

• The end-to-end model includes GCN, transitivity-probability, and converse-probability parameters, and the experiments focus on WSGC-S.The layout-to-image component is generally a pretrained LostGAN; CLEVR uses a trained AttSPADE model.Source: §5 · pp. 8–9
• Training minimizes L1 error between predicted and ground-truth bounding boxes, with the loss evaluated over sampled WSGC-S completions.The model samples independent converse-relation variables for the input graph.Source: §5 · p. 8
• The converse sampling distribution depends on its parameters, so the paper applies REINFORCE to obtain the loss gradient for those parameters.One sampled completion is used per image in practice.Source: §5 · p. 8

6 Experiments

• The evaluation covers synthetic data, COCO-stuff, Visual Genome, and CLEVR; layout metrics include mIOU and recall at IOU thresholds 0.3 and 0.5.The experiments assess both scene-graph-to-layout performance and image generation from the resulting layouts.Source: §6 · pp. 9–10, §6.1 · p. 10

6.1 Scene-Graph-to-layout Generation

• On synthetic data, learned transitivity closely matches known transitivity, while the baseline performs worse and degrades when tested on up to 128 objects after training on 16.The baseline requires more GCN layers as object count grows, whereas WSGC performs well with two layers.Source: §6.1 · pp. 10–11, Figure 6 · p. 11
• In the standard COCO/VG setting, WSGC has comparable performance to baselines, while increasing Sg2Im’s GCN size results in overfitting.The standard setting contains relatively few objects.Source: §6.1 · p. 11
• WSGC improves predicted layouts on all reported metrics in packed COCO and VG scenes with 16 or more objects.The packed setting evaluates training and test images with at least 16 objects.Source: §6.1 · p. 11, Table 1 · p. 12
• For semantically equivalent and noisy Packed COCO graphs, WSGC outperforms baselines and better handles inputs with 10% randomly modified relations.Table 2 evaluates semantically equivalent test graphs and noisy graphs with 10% randomly chosen relations.Source: Table 2 · p. 12, §6.1 · p. 12

6.2 Scene-graph-to-image Generation

• On packed datasets, WSGC improves generated-image quality on COCO and VG; on CLEVR it outperforms Sg2Im in IOU and is ranked higher in 96.8% of comparisons.For Packed COCO and VG, the image-generation comparison uses LostGAN; CLEVR uses AttSPADE.Source: Table 3 · p. 13, §6.2 · p. 13

7 Conclusion

• The method maps scene graphs to images invariant to a set of logical equivalences.The conclusion frames this as preprocessing a scene graph before layout generation.Source: §7 · p. 14
• Canonical representations improve layouts and image quality, and handle packed scenes with fewer layers than non-canonical approaches.The authors attribute the layer reduction to closure calculation propagating information across the graph.Source: §7 · p. 14
• The study covers two types of relation properties; extending closure to more complex properties may be computationally hard and amount to inference in Markov Logic Networks.The authors describe further invariance modeling as a direction for future work.Source: §7 · p. 14

1.1 Exact Weighted Scene Graph Canonicalization (WSGC-E)

• WSGC-E adds converse edges using learned converse weights and transitive edges using the maximum path weight multiplied by the transitivity probability.The maximum path weight can be found with a shortest-weight-path algorithm, but dense completion makes training and inference slow.Source: §1.1 · pp. 19–20, Figure 9 · p. 20

1.2 Empirical Comparison of WSGC-E and WSGC-S

• WSGC-E and WSGC-S achieve comparable performance on standard COCO and Visual Genome, while WSGC-E cannot run on packed datasets with 24,000 triplets per image.Standard-dataset graphs average fewer than 1,000 triplets per image; WSGC-S adds fewer edges and is the practical alternative for packed graphs.Source: §1 · p. 19, §1.2 · p. 21, Table 4 · p. 21

1.4 Weighted Graph Convolutional Network

• The weighted GCN aggregates incident-edge messages using canonical-graph weights and predicts each object’s layout from its final representation.It uses learned object and relation embeddings and iteratively updates representations.Source: §1.4 · pp. 21–22

1.5 Generalization on Packed Scenes

• Larger Sg2Im models did not explain WSGC’s packed-scene gains; increasing model capacity instead suggests overfitting, supporting canonicalization as the source of improvement.The tested variants included 32- and 64-layer models and a model with 1024 hidden units and 16 layers.Source: §1.5 · p. 22

2 Layout-to-Image with AttSPADE

• For CLEVR, the paper introduces AttSPADE as a generator that directly controls attributes of generated images, a capability not supported by generators such as LostGAN.The authors describe the generator as an additional contribution rather than the main focus.Source: §2 · p. 23

2.1 The AttSPADE Model

• AttSPADE conditions generation on object attributes rather than only object class, representing each box’s attributes in a tensor supplied to a SPADE generator.A multi-hot attribute vector is projected to an embedding that is repeated across the box’s pixels.Source: §2.1 · p. 23

2.2 The Loss Functions

• The generator uses a multi-scale discriminator with hinge loss, plus a box discriminator trained on cropped real and generated object images.The image discriminator follows SPADE.Source: §2.2 · p. 24

2.3 Baseline Models

• The comparison varies both layout source and image generator, including ground-truth, WSGC-predicted, and Sg2Im layouts and a Grid2Im variant without coarse ground-truth layout information.The generator choices include AttSPADE, LostGAN, and Grid2Im.Source: §2.3 · p. 24

2.4 Results

• AttSPADE improves over previous approaches from ground-truth layouts at both resolutions, while the end-to-end model outperforms most baselines on COCO and Visual Genome.The comparison reports Inception Score, FID, and Diversity Score; qualitative results also describe higher-quality images and WSGC-layout images similar to those from ground-truth layouts.Source: §2.4 · pp. 24–25, Table 5 · p. 24

3.1 Synthetic dataset

• The synthetic dataset tests transitivity closure using square objects and spatial relations, with scene-graph/layout pairs generated from sampled object centers and sizes.“Above” is transitive, while “Opposite Horizontally” and “XNear” are not.Source: §3.1 · p. 25

3.2 Packed Datasets

• Packed datasets include images with at least 16 objects; Visual Genome receives additional geometric relations, with redundant edges removed to form minimal graphs.The construction yields graphs described as more complex and informative than those in; the figure illustrates relation labels and object-size annotations.Source: §3.2 · pp. 25–26, Figure 12 · p. 26

4.1 Scene-Graph-to-layout

• The weighted GCN uses five hidden layers, 128-unit object and relation embeddings, and three-layer 512-unit MLPs for its update functions.Adam learning rates are 1e−2 for converse and transitive parameters and 1e−4 for the other parameters.Source: §4.1 · p. 26

4.2 AttSPADE

• AttSPADE uses spectral normalization, Adam with β1 = 0.5 and β2 = 0.999, and a 0.0001 learning rate for both generator and discriminator.Experiments use NVIDIA V100 GPUs, synchronized BatchNorm, and batch sizes of 32 at 128 × 128 resolution and 16 at 256 × 256 resolution.Source: §4.2 · p. 26

• SGC outputs exactly the closure C(E), as established by proving both set inclusions.Source: §5 · pp. 26–27
• For ˆC(E) ⊆ C(E), original edges and converse-constructed edges belong to C(E), while transitive edges follow from paths whose edges belong to the closure.Source: §5 · p. 27
• For C(E) ⊆ ˆC(E), each implied relation is retained if original, otherwise its path or converse relation entails its inclusion in ˆC(E).Source: §5 · p. 27

• Table 2 demonstrates that learned WSGC is more robust to changes in scene-graph input after test graphs are transformed into semantically equivalent variants.Source: §6 · p. 27
• The variants drop one of each converse-relation pair with probability 0.5, then remove semantics-preserving transitive-closure edges with probability 0.5.Source: §6 · p. 27
• On COCO-Stuff, selected figures compare AttSPADE with Grid2Im and LostGAN from GT masks at 128 × 128, and compare layout- and graph-based generation at 256 × 256.Source: Figure 13 · p. 28, Figure 14 · p. 29
• On Visual Genome, selected figures compare AttSPADE and LostGAN from GT boxes or scene graphs at 128 × 128, and show AttSPADE generation from layouts or graphs at 256 × 256.Source: Figure 15 · p. 30, Figure 16 · p. 31