PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Visual Commonsense for Robust Scene Graph Generation

Alireza Zareian, Zhecan Wang, Haoxuan You, Shih-Fu Chang

arXiv:2006.09623v2cs.CVcs.LG

TL;DR

Scene graph generation can produce implausible compositions when perception errors conflict with real-world patterns. The paper learns structured visual commonsense from scene graphs with a Global-Local Attention Transformer and fuses it with perception to improve robustness and scene-understanding accuracy.

  • Problem

    Scene graph generation models are vulnerable to perception errors that yield nonsensical object–predicate compositions, while existing corrections rely on external knowledge or simple dataset statistics.

  • Method

    The method learns structured visual commonsense automatically from a scene graph corpus using a Global-Local Attention Transformer, then fuses separate perception and commonsense models based on confidence.

  • Results

    The method outperforms conventional transformers and graph-based networks, learns commonsense without external knowledge or dataset bias, and improves downstream scene understanding.

  • Takeaways & Limitations

    Separating commonsense reasoning from visual perception and combining them under uncertainty produces more robust scene graph generation.

  • Takeaways & Limitations

    The commonsense model may favor statistically common structures and fail to predict less common structures because autoencoders under-represent long-tailed distributions.

Abstract

from arXiv · show

Scene graph generation models understand the scene through object and predicate recognition, but are prone to mistakes due to the challenges of perception in the wild. Perception errors often lead to nonsensical compositions in the output scene graph, which do not follow real-world rules and patterns, and can be corrected using commonsense knowledge. We propose the first method to acquire visual commonsense such as affordance and intuitive physics automatically from data, and use that to improve the robustness of scene understanding. To this end, we extend Transformer models to incorporate the structure of scene graphs, and train our Global-Local Attention Transformer on a scene graph corpus. Once trained, our model can be applied on any scene graph generation model and correct its obvious mistakes, resulting in more semantically plausible scene graphs. Through extensive experiments, we show our model learns commonsense better than any alternative, and improves the accuracy of state-of-the-art scene graph generation methods.

1 Introduction

The paper learns visual commonsense directly from scene graphs and separates it from perception, then fuses both signals to correct implausible scene-graph predictions. Its GLAT model acquires structured commonsense and improves robust scene understanding across perception conditions.

  • Motivation: Existing scene graph generation models can make perceptual errors that produce implausible object–predicate compositions, which commonsense can correct.A bird misclassified as a bear can be corrected because bear on branch conflicts with intuitive physics or animal behavior.
  • Contribution: The paper learns graphical commonsense automatically from annotated scene graphs without external knowledge, capturing structured patterns beyond co-occurrence heuristics.The approach is framed as auto-encoding perturbed scene graphs and predicting masked scene-graph elements.
  • Architecture: Perception and commonsense are disentangled into separately trained models, whose disagreement is resolved by a fusion module using both graphs and their confidence values.The fusion module can favor commonsense when perception is uncertain and perception when visual evidence is clear.
  • Contribution: GLAT extends Transformers with local attention heads so the model can encode scene-graph structure while learning commonsense from a scene-graph corpus.Local heads attend to graph neighbors, complementing global attention over all nodes.
  • Results: Experiments show GLAT learns affordance and intuitive physics, remains robust to dataset bias in rare and zero-shot scenarios, and improves downstream scene-graph generation.The GLAT and Fusion mechanism can be applied to any SGG method to correct mistakes and improve accuracy.

2 Related Work

Prior vision methods obtain commonsense from external knowledge bases, corpus statistics, or specialized models, but these sources are incomplete, heuristic, or narrow. This paper instead formulates visual commonsense learning over structured scene graphs and uses global-local attention to represent their relationships.

  • Commonsense in computer vision: Commonsense has been incorporated into object recognition, detection, segmentation, action recognition, relation detection, scene-graph generation, and visual question answering.The related work organizes these methods by where commonsense comes from and how models use it.
  • Commonsense sources: Existing approaches commonly use curated knowledge bases or corpus statistics, which are limited by incompleteness, hard-coded heuristics, or co-occurrence bias.ConceptNet is cited as an external source, while frequency statistics provide ad-hoc alternatives.
  • General framework: Specialized models address individual phenomena such as intuitive physics or object affordance, whereas this work proposes a more general scene-graph framework.Scene graphs provide a semantic representation that can include relationships beyond object co-occurrence.
  • Commonsense in SGG: Earlier scene-graph methods integrate commonsense through frequency priors, message propagation, retrieved facts, or auxiliary training objectives.Frequency priors can favor frequent relations such as wearing over visually predicted holding.
  • Proposed approach: The proposed method learns commonsense as a machine-learning task using a graph-based neural network rather than relying on a limited external knowledge source.Its Transformer variant replaces some global attention heads with local attention over graph neighbors.

3 Method

The method separates perception from visual commonsense, models commonsense by reconstructing perturbed scene graphs, and fuses both predictions according to their confidence. Its GLAT architecture combines global and relation-specific local attention to encode scene-graph structure without image input.

  • Scene-graph representation: The scene-graph representation treats predicates as nodes connected to subject and object entities, allowing multiple predicates between the same entity pair and unified attention over all nodes.This representation differs from the conventional edge-based formulation while remaining convertible to it.
  • Framework: The framework approximates MAP scene-graph inference by combining an image-based perception model with an image-independent commonsense model.The perception model produces GP, while the commonsense model produces GC as a more plausible graph.
  • Commonsense model: The decoder reconstructs the correct graph from perturbed node embeddings without access to the image, keeping the commonsense block separate from perception.The figure depicts only this commonsense block rather than the full pipeline.
  • Commonsense model: The commonsense model learns a generative distribution over scene structures by denoising perturbed annotated graphs with a graph-based autoencoder.Training adds noise to Visual Genome scene graphs, reconstructs nodes and edges, and uses separate cross-entropy losses for both classifiers.
  • Global-Local Attention Transformers: GLAT augments Transformer self-attention with local heads so node embeddings encode global context and scene-graph neighborhoods.Global heads attend to all nodes, whereas local heads attend only to immediate neighbors and can distinguish subject and object edge types.
  • Fusion: The fusion module combines perception and commonsense predictions using their classification confidences, assigning more influence to the model judged more reliable for each image.It computes weighted fused logits for corresponding nodes and applies a final softmax distribution.

4 Experiments

Experiments evaluate GLAT’s commonsense acquisition, its learned affordance, physics, and composition patterns, and its effect on scene graph generation. The results show improved reconstruction and robust fusion-based correction of perception errors.

  • Evaluation setup: The experiments evaluate GLAT through masked scene graph reconstruction, commonsense prediction statistics, downstream scene graph generation, and qualitative correction examples.The evaluation uses Visual Genome and compares perception, commonsense, and fused outputs.
  • Commonsense acquisition: 30% node masking evaluates how accurately GLAT reconstructs masked entity and predicate classes from global context and local graph structure.The evaluation reports entity, predicate, and overall accuracy under graph reconstruction.
  • Commonsense acquisition: GLAT significantly outperforms global-only transformers, local-only graph attention, graph convolutional networks, and a frequency prior on semantic graph reconstruction.The baselines represent transformer, graph-attention, graph-convolutional, and frequency-based alternatives.
  • Learned commonsense: Prediction statistics show plausible affordance, intuitive physics, and object-composition patterns, including rare or unseen combinations that contrast with IMP’s nonsensical predictions.Examples include person–horse interactions, plausible objects under a bed, and objects having ears.
  • Scene graph generation: The full perception–commonsense fusion model consistently improves scene graph generation, while GLAT alone improves IMP but does not significantly change SNM and KERN.Evaluation uses mean recall at 50 and 100 for SGCls and PredCls.
  • Qualitative analysis: Qualitative examples show the commonsense model correcting perception errors when intuitive physics makes a prediction unlikely, after which fusion accepts the correction.Figure examples merge outputs from perception, commonsense, and fusion modules and mark mistakes against ground truth.

5 Conclusion

The paper concludes that GLAT learns structured visual commonsense from scene graph corpora and complements perception through confidence-based fusion. Experiments support improved robustness in scene graph generation.

  • 5 Conclusion: The method learns structured commonsense patterns through self-supervised training rather than simple co-occurrence statistics.It augments transformers with local attention heads and outperforms transformer and graph-based alternatives.
  • 5 Conclusion: Separate perception and commonsense models complement each other under uncertainty through fusion based on their confidence values.The fusion mechanism determines when to trust perception and when to use commonsense.
  • 5 Conclusion: Experiments show the method is effective for scene graph generation and motivate applying the methodology to other computer vision tasks.The stated conclusion concerns the demonstrated scene graph generation setting and proposed future direction.

Supplementary Material

The supplementary material adds implementation details, a prediction-statistics evaluation protocol, and qualitative examples of learned commonsense.

  • Supplementary Material: The supplementary material provides implementation details omitted from the main paper because of limited space.It begins with additional implementation information before presenting further analysis.
  • Supplementary Material: It introduces an evaluation protocol that reveals statistics about the data and state-of-the-art performance.The protocol is presented alongside analysis of model predictions.
  • Supplementary Material: It provides qualitative examples to showcase what the commonsense model learns.These examples follow the implementation and evaluation analyses.

A Implementation Details

Training proceeds in three stages: perception, independent commonsense learning, and joint fine-tuning. GLAT is trained on masked Visual Genome graphs and then adapts perception outputs while preserving their graph structure.

  • A Implementation Details: The training pipeline has perception training, separate GLAT commonsense training, and joint fine-tuning stages.GLAT is trained once independently before being stacked on each perception model.
  • A Implementation Details: GLAT uses six layers, eight attention heads per layer, 300-D representations, 30% masking, Adam, learning rate 0.0001, and 100 epochs.These settings apply to training on Visual Genome scene graphs.
  • A Implementation Details: For adaptation, the system keeps the top 100 perception triplets, feeds the resulting graph to GLAT without masking, and fine-tunes for 25 epochs at learning rate 0.00001.The decoder changes node classification logits while the graph structure is kept fixed.
  • A Implementation Details: The decoder classifies each node and node pair to reconstruct entity, predicate, and edge outputs after noise is added to annotated graphs.The encoder and decoder are trained end-to-end on noisy Visual Genome scene graphs.

B Quantitative Evaluation

The evaluation revisits recall-based SGG metrics because Visual Genome’s imbalanced triplet frequencies can make overall and predicate-level mean recall misleading. It proposes logarithmic frequency bins to assess performance across rare and frequent triplets.

  • Evaluation limitations: Overall recall can overstate usefulness because Visual Genome ground-truth triplets have highly disproportionate statistics.Mean recall reduces predicate-frequency dominance by averaging recall across predicate classes, but does not resolve imbalance among compositions within each class.
  • Evaluation limitations: Mean recall still favors common compositions within a predicate bin, such as person holding racket over person holding cellphone.The imbalance arises because triplets with the same predicate can have very different frequencies.
  • Proposed evaluation: Logarithmic bins using powers of 3 provide the proposed strategy for balancing the number of frequency groups.The authors selected this strategy by trial and error to avoid having too few or too many bins.
  • Frequency-stratified results: The rarest bin contains triplets appearing 1–3 times in training and represents 14.7% of VG test triplets, or 75.6% of unique triplets.The most frequent bin contains the 4 most frequent triplets, representing 3.1% of the test set and 0.00004% of unique triplets.
  • Frequency-stratified results: State of the art recall is 16.7% on the rarest bin versus 77.1% on the most frequent bin, exposing severe performance disparity.This disproportion motivates evaluation that emphasizes rare but plausible situations rather than a few frequent triplets.

C Qualitative Results

Qualitative examples show GLAT correcting perception errors through object affordances, contextual relations, and intuitive physics, while fusion selectively trusts commonsense or perception based on confidence. The examples also include a case where fusion rejects an incorrect commonsense correction.

  • Examples of corrections: GLAT corrects giraffe-head, train-wheel, vase-flower, and pizza-box relations that perception misclassifies.These corrections use contextual plausibility, object function, and learned distinctions between physical location and containment.
  • Fusion behavior: Fusion prefers commonsense for the giraffe and train examples because its predictions have higher confidence.The system combines perception and commonsense graphs rather than always replacing visual predictions.
  • Abstract and ambiguous relations: Commonsense distinguishes holding from using and wearing pants from having a leg when visual features do not resolve the relation.The model abstracts interactions into symbolic concepts and can use past experience without relying directly on image clarity.
  • Failure case: Fusion retains perception when commonsense incorrectly changes leg of bench to leg on bench.The visually obvious placement of the bench legs lets fusion reject the commonsense model’s typical but incorrect alternative.
Loading 2006.09623v2…