PaperlayerView exact revision on arXiv

Source-linked AI summary

Situation Recognition with Graph Neural Networks

Ruiyu Li, Makarand Tapaswi, Renjie Liao, Jiaya Jia, Raquel Urtasun, Sanja Fidler

arXiv:1708.04320v1cs.CV

TL;DR

Situation recognition seeks to predict an image’s salient verb and the nouns filling its semantic roles, a difficult task given varied role values and image contexts. The paper uses a Graph Neural Network to propagate information among roles and model their dependencies. It reports an ∼4.5% accuracy improvement on value-all over prior work.

  • Problem

    Situation recognition requires predicting a verb and its role-noun pairs, but role types and values are numerous and vary across images of the same verb.

    Source: §1 · p. 1

  • Method

    A Graph Neural Network propagates messages recurrently between verb and role nodes to model dependencies and produce a structured situation prediction.

    Source: §1 · pp. 1–2

  • Results

    ∼4.5% accuracy improvement on value-all was achieved on the imSitu benchmark dataset.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors’ analysis demonstrates the need to capture dependencies between roles, allowing nouns to inform each other.

    Source: §5 · p. 8

    Synonymous nouns and verbs make the task challenging even though each image comes with three annotations.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

We address the problem of recognizing situations in images. Given an image, the task is to predict the most salient verb (action), and fill its semantic roles such as who is performing the action, what is the source and target of the action, etc. Different verbs have different roles (e.g. attacking has weapon), and each role can take on many possible values (nouns). We propose a model based on Graph Neural Networks that allows us to efficiently capture joint dependencies between roles using neural networks defined on a graph. Experiments with different graph connectivities show that our approach that propagates information between roles significantly outperforms existing work, as well as multiple baselines. We obtain roughly 3-5% improvement over previous work in predicting the full situation. We also provide a thorough qualitative analysis of our model and influence of different roles in the verbs.

1. Introduction

Situation recognition predicts an image’s salient verb together with a frame of role-noun pairs, a challenging task because roles and their possible values vary widely across verbs and image contexts. The paper proposes a Graph Neural Network that propagates information among roles to model their dependencies.

1. Introduction

  • Situation recognition predicts a verb and its frame of role-noun pairs, adding details such as who performs an action and which tool is used.The paper motivates this richer understanding with examples such as identifying the person repairing a house and the tool used.Source: §1 · p. 1, Figure 1 · p. 1
  • The task is challenging because the numbers of actions, role types, and possible role values are large, and images of the same verb can contain very different contexts and nouns.The figure caption also notes variation in role content across images with the same verb.Source: §1 · p. 1, Figure 2 · p. 2
  • The prior CRF approach modeled dependencies between verbs and role-noun pairs but did not explicitly model dependencies between roles.The paper positions role-to-role modeling as its focus.Source: §1 · p. 1
  • GNN message propagation models dependencies between roles and produces a consistent structured output, significantly improving performance over existing work and strong RNN baselines.The paper explores different connectivity structures among role nodes.Source: §1 · pp. 1–2

2. Related Work

Prior work covers situation recognition, visual semantic role labeling, and broader image-understanding tasks. This paper distinguishes its approach by explicitly modeling dependencies between roles with neural architectures.

2. Related Work

  • Situation-recognition work includes scene graphs, visual semantic role labeling, and imSitu frames; the CRF baseline jointly predicts verbs and role-noun triplets, while other work shares nouns across roles or augments training data.The paper’s focus differs by explicitly modeling dependencies between roles for each verb.Source: §2 · p. 2
  • Visual question answering and image captioning use CNN-RNN pipelines that typically learn verb-noun dependencies implicitly, while scene graphs represent objects, attributes, and relationships.Scene graphs have also been used for visual relationship detection and image-caption generation.Source: §2 · p. 2
  • Visual semantic role labeling builds on NLP semantic role labeling and resources such as FrameNet and WordNet to describe actions with verb-role-noun pairs.Source: §2 · p. 2
  • Graph-neural-network approaches include graph convolutions and recurrent node updates; GGNNs use gated updates learned with back-propagation through time, and this paper applies GGNNs to situation recognition.The related work also notes graph-neural approaches using LSTM-based update functions and applications to image classification.Source: §2 · p. 2

3. Graph-based Neural Models for Situation Recognition

Situation recognition predicts an image’s verb and the nouns filling its corresponding semantic roles. The paper models dependencies among these outputs with graph-based neural models and compares them with simpler sequential and tree-structured RNNs.

3. Graph-based Neural Models for Situation Recognition

  • A situation is a verb paired with a realized frame of role–noun values, with unknown or inapplicable values represented by ∅.For example, riding can be paired with agent:dog, vehicle:surfboard, and place:sidewalk.Source: §3 · pp. 2–3

3.1. Graph Neural Network

  • The GGNN represents each image with a verb node and one node per frame role, connecting all role pairs with undirected edges to model their dependencies.Node states are initialized from verb and noun CNN image features together with verb and role embeddings, then normalized before propagation.Source: §3.1 · p. 3
  • At each propagation step, nodes aggregate messages from neighboring hidden states and update their states through a GRU-like gate before outputs predict the verb and role nouns.At inference, the predicted verb selects a frame; the model then propagates among its role nodes and selects a noun for each role.Source: §3.1 · pp. 3–4

3.2. Simpler Graph Architectures

  • The chain RNN predicts the verb first, then agent and place, followed by other roles ordered by their frequency across verb frames.Unlike the GGNN, chain nodes receive information once from their left neighbor and predict immediately; the role order must be specified manually.Source: §3.2 · p. 4
  • The tree-structured RNN places the verb at the root, with agent and place preceding the other roles so they have a greater chance to influence their predictions.It uses the same GRU update and learning and inference procedures as the chain RNN.Source: §3.2 · p. 4
  • 1.9% performance drop follows changing the chain order from verb → agent → place to verb → agent → place → … .The paper notes that predicting place requires a more global view of the image than predicting agent.Source: §3.2 · p. 4

4. Evaluation

The evaluation tests graph-based situation prediction on imSitu, scoring verb-role-noun tuples and complete situations. Results favor role information sharing, while qualitative analyses show both useful role interactions and annotation-related mismatches.

4. Evaluation

  • Evaluation uses imSitu’s 75k/25k/25k splits and verb, value, and value-all metrics; two fine-tuned VGG-16 CNNs provide verb and noun features.The noun CNN predicts the top 2,000 most frequent nouns, covering about 95% of nouns.Source: §4 · p. 5, §4.1 · p. 5

4.2. Results

  • In FC Graph, propagation improves all metrics by a mean 3.8% and gains saturate soon as T increases; beam search with B=10 adds a small improvement.The authors attribute early saturation to the fully connected graph sharing most information in the first propagation step.Source: §4.2 · pp. 5–6
  • Top-1 predictions achieve the best performance on all metrics; value-all is 3–4% higher than [47] despite its data augmentation.Table 3 compares development and test results against state-of-the-art models; the value-all comparison is for top-1 predicted verbs.Source: Table 3 · p. 6, §4.2 · p. 7
  • Ground-truth verb embeddings raise top-1 value-all from 17.70% to 19.15%; tanh versus ReLU has negligible impact, and predicting all annotations slightly beats Soft-OR.The model uses tanh for its experiments.Source: §4.2 · p. 6
  • Connected roles matter: Chain RNN improves top-1 value-all by 3.9% and Tree RNN by 2%; the chain needs T=8 versus T=6 for tree and T=4 for FC Graph.The authors note that chain connectivity helps place and agent exchange information, while joint noun prediction can reduce verb performance.Source: §4.2 · pp. 6–7

4.3. Further Discussion

  • Propagation norms show tool influencing other roles for fastening, while agent and obstacle influence roles for jumping.The displayed matrices summarize incoming-message norms averaged across development images at the first time step.Source: §4.3 · p. 7, Figure 7 · p. 7
  • Some wrong verb predictions are plausible alternatives, such as sitting versus slouching, and can still yield correct role-noun pairs despite being scored wrong.Choosing another verb can also select a different set of roles.Source: §4.3 · p. 7, Figure 8 · p. 7
  • With correct verbs, errors favor majority-class outdoor and frequent apple predictions; reasonable predictions can also miss ground truth because synonymous nouns and verbs remain challenging.The paper notes apple appears 79 times versus cotton’s 14 and suggests more training samples could help; some examples are reasonable but unannotated.Source: §4.3 · p. 8

5. Conclusion

The paper presents a GNN for predicting image situations as verbs and role-noun pairs, explicitly modeling dependencies so nouns can inform one another. On imSitu, it reports a roughly 4.5% improvement in value-all accuracy and analyzes role dependencies alongside RNNs and related approaches.

  • The GNN explicitly models dependencies between verbs and roles, allowing role nouns to inform one another in situation recognition.The task predicts a verb with its corresponding frame of role-noun pairs.Source: §5 · p. 8
  • On imSitu, the model improves value-all accuracy by roughly 4.5%.Value-all evaluates correctness of the entire frame.Source: §5 · p. 8
  • The paper analyzes the need to capture role dependencies and compares its approach with RNN models and related solutions.Source: §5 · p. 8

Supplementary Material

The supplementary material analyzes verb prediction and visualizes learned verb and role embeddings. It also examines role influence on verb-frame prediction and provides example model predictions.

  • The supplement analyzes verb prediction performance.The analysis appears in Section A.Source: Supplementary Material · p. 11
  • It uses t-SNE plots to visualize learned verb and role embeddings.The embedding visualizations are presented in Section B.Source: Supplementary Material · p. 11
  • It examines how different roles influence verb-frame prediction and presents example predictions.Role influence is visualized through propagation matrices in Section C; examples appear in Section D.Source: Supplementary Material · p. 11

A. Verb Prediction

Verb prediction accuracy varies across verbs, and errors between visually similar verbs contribute to incorrect situation recognition. The supplement reports accuracy patterns and groups confusion matrices by FrameNet verb clusters.

  • About 22% of verbs are classified correctly over 50% of the time, while attaching, making, and placing have accuracy below 10%.Random performance is close to 0.2% across 504 verbs.Source: A. Verb Prediction · p. 11, Figure 10 · p. 11
  • Verb prediction matters because an incorrect verb makes the whole situation count as incorrect, even when role-noun predictions improve through shared information.Source: A. Verb Prediction · p. 11
  • The confusion matrices group verbs by FrameNet clusters, with predictions outside each cluster collected under “others.”The dataset's 504 verbs are grouped into 161 FrameNet verbs; 73 clusters contain only one verb.Source: A. Verb Prediction · p. 11, Figure 11 · p. 12
  • Similar verbs are significantly confused, including visually hard-to-differentiate pairs such as colliding, crashing, and ramming.Crying, giggling, laughing, and weeping are another cited confusion group.Source: A. Verb Prediction · p. 11, Figure 11 · p. 12

B. Verb and Role Embeddings

The model initializes role-node hidden states using image features and embeddings for the role and predicted verb. The learned verb embeddings form clusters, including associations across FrameNet groups, while related role embeddings also appear close together.

  • Role-node initialization uses the image representation together with role and predicted-verb embeddings, with verbs and roles embedded in R1024 space.The image representation comes from the noun-prediction CNN.Source: B. Verb and Role Embeddings · p. 12

Verbs

Roles

  • Related role embeddings appear close together, both for role pairs unique to one verb and for semantically related roles such as food, heatsource, and container.The visualizations include the learned role embeddings in two dimensions.Source: B. Verb and Role Embeddings · pp. 12–13

C. Visualizing the propagation matrices.

The supplement extends the propagation-matrix visualization to 15 randomly selected verbs. Many verbs have dominant roles that influence others; the matrices display message norms between roles.

C. Visualizing the propagation matrices.

D. Prediction Results

The examples show fully correct predictions, correct verbs with some incorrect role nouns, and plausible predictions with an incorrect verb. They illustrate what the prediction metrics capture and miss.

D. Prediction Results

  • Figure 16 shows correct verb predictions with some incorrect role nouns, counted by value but not value-all.Some apparent errors arise because the ground-truth annotations do not match the depicted image.Source: D. Prediction Results · p. 19, Figure 16 · p. 21
  • Figure 17 shows plausible top-scoring predictions with an incorrect verb; their often-correct role-noun pairs are not captured by any metric.Some verbs share a frame and role set, allowing the model to infer matching roles despite the verb mismatch.Source: D. Prediction Results · p. 19, Figure 17 · p. 22
  • Figure 15 shows examples where the predicted verb and every role-noun pair are correct.These are development-set top-1 predictions.Source: D. Prediction Results · p. 19, Figure 15 · p. 20
Loading 1708.04320v1…