Grounded Situation Recognition

• Situation recognition produces structured descriptions with semantic roles, unlike image captioning, and avoids captioning’s evaluation challenges.Source: §1 · p. 2
• GSR identifies the salient situation and grounds its roles in the image, addressing challenges including semantic sparsity, ambiguity, scale, and occlusion.Situation recognition describes the activity, actors, objects, and their roles; GSR adds where the involved entities are located.Source: §1 · p. 2
• SWiG adds 278,336 bounding-box groundings to imSitu frames, covering most of its more than 10k entity classes and exhibiting a long-tailed class distribution.The diversity of activities, images, and grounded classes makes the dataset challenging for existing approaches.Source: §1 · p. 2
• The authors modify RetinaNet to scale to roughly 10k categories, then build ISL, which predicts situations and groundings independently and combines them with late fusion.JSL is the proposed joint alternative to ISL.Source: §1 · pp. 2–3
• JSL improves both semantic role prediction and grounding, with 8% to 32% relative gains over ISL across the entire suite of grounding metrics.JSL jointly predicts situations and groundings conditioned on image context, with gradients backpropagated through the entire network during training.Source: §1 · p. 3
• GSR enables conditional querying about a specified image region, visual chaining of higher-order activity relations, and image retrieval conditioned on linguistic or visual inputs.Conditional Situation Localizer (CSL) produces grounded situations given an image and a region of interest.Source: §1 · p. 3

Describing Activities in Images.

• Early image-activity work used few verbs or verb-object tuples, whereas imSitu defines a broad, balanced verb and role space using linguistic resources.Captioning datasets and neural models are established [9], but evaluation remains difficult despite proposed metrics; structured situation frames have clear metrics.Source: Describing Activities in Images · pp. 3–4

Visual Grounding.

• Compared with human-centric grounding datasets, SWiG spans 504 verbs, up to six roles, nearly 10,000 objects, and actions by animals and objects.Visual grounding links caption words to image regions, while referring-expression work identifies regions unambiguously; RefClef, RefCOCO, RefCOCO+, RefCOCOg, and GuessWhat?! are examples. Unlike SWiG’s salient activities and verb roles, Visual Genome annotates dense graphs whose top ten relations are mostly part or positional (66%); missing relations make evaluation tricky.Source: Visual Grounding · p. 4

Situation Recognition Models.

• Situation-recognition models progressed from CRFs and specialized predictors to role-dependency and attention graph networks; grounded models build on an RNN approach while remaining applicable to alternatives.The grounded approach builds on Mallya et al.’s RNN model; its RetinaNet modifications borrow ideas from large-class detectors including YOLO-9000, DLM-FA, R-FCN-3000, and CS-R-FCN [18].Source: Situation Recognition Models · pp. 4–5

3 GSR and SWiG

• GSR predicts a visually groundable verb, its semantic role values, and bounding-box groundings for entities in the image.The verb set contains 504 classes; frames have 1–6 roles, and a role may be absent or labeled without a grounding, such as when its entity is occluded.Source: Task · p. 5

Task.

• SWiG retains imSitu’s images, frame annotations, and splits, with 126,102 images and three annotated frames per image across 504 verbs.Each frame has 1–6 roles and averages 3.55 roles.Source: Data · pp. 5–6

Data.

• 278,336 of SWiG’s 435,566 non-∅ noun slots (63.9%) have bounding boxes, combined from three workers’ annotations.Boxes are absent when objects are not visible or for the Place role, which is never boxed because it refers to the whole image.Source: Data · p. 6
• SWiG has a long-tailed distribution of grounded nouns, from “man” appearing over 100,000 times to nouns appearing once.Unlike MS-COCO, its grounded-object frequencies resemble a long-tail distribution.Source: Data · p. 6, Figure 3 · p. 6
• Grounding frequency varies by role: Agent is common, Firearm is rare, and Place is never grounded; Event roles usually lack a specific visible object.Roles such as Firearm and Teacher are nearly always visible when relevant, while role frequencies also reflect the frame’s coverage of concrete and abstract concepts.Source: Data · p. 6
• Verb labels provide priors for entity scale and aspect ratio, but appearance alone may not distinguish activities such as Mowing and Harvesting.For example, ropes associated with dragging or pulling tend to be horizontal, while ropes in hoisting or climbing situations tend to be vertical; noun grounding scale and aspect ratio also vary across instances.Source: Data · pp. 6–7, Figure 4 · p. 7

Situation Recognition Model.

• The situation recognizer uses ResNet-50 embeddings to predict the verb and an LSTM to sequentially predict role nouns in dataset order.Its loss sums label-smoothed noun cross-entropy and verb cross-entropy. The paper selects the RNN without fusion for its accuracy, simple architecture, and ability to reimplement and tune it; code and models for attention graph nets were unavailable.Source: Situation Recognition Model · p. 8

Large-Class-Cardinality Object Detection

• A modified RetinaNet detector scales object detection to SWiG’s 10,000 noun classes.It predicts objectness, selects and featurizes the top 100 boxes with RoI Align, and classifies them into noun categories; it also drops the finest FPN scale and clusters anchor aspect ratios on training data. The modifications enable batch size 64 on four 24-GB TITAN RTX GPUs.Source: Large-Class-Cardinality Object Detection · pp. 8–9

Independent Situation Localizer (ISL)

• ISL independently runs situation recognition and detection, then assigns each predicted role noun its highest-scoring box.A role remains ungrounded if all box scores for its noun fall below a threshold or if the role is Place.Source: Independent Situation Localizer (ISL) · p. 9

Joint Situation Localizer (JSL)

• JSL jointly predicts a situation and its object locations, conditioning each role’s noun and grounding on the verb and previous roles.Its LSTM uses verb, global image, previous-noun, and previous-box features; localization is conditioned on the current role, and the model explicitly scores whether an object is grounded. It localizes one noun per step, marking relevant anchors positive only when that noun is visible and grounded in training data.Source: Joint Situation Localizer (JSL) · pp. 9–10

Implementation Details.

• ISL and JSL use two ResNet-50 backbones and have approximately 108 million parameters each; training uses Adam on four 24GB TITAN RTX GPUs for about 20 hours.Adam uses momentum parameters β = (0.9, 0.999).Source: Implementation Details · p. 10

Metrics.

• Evaluation averages five metrics across verbs: verb, value, value-all, grounded-value, and grounded-value-all; grounding requires IoU of at least 0.5.Metrics are reported with the ground-truth verb, top-1 predicted verb, or top-5 verb accuracy; noun and grounding predictions are incorrect when the top-1 verb is wrong, and are conditioned on the correct verb for top-5.Source: Metrics · p. 10

Results.

• JSL improves over ISL across every metric at equal parameter count, with grounding gains from 8.6% on Ground-Truth-Verb–ground-value to 32.9% on Top-1-Verb–grounded-value-all.Table 1 reports SWiG dev-set evaluation.Source: Table 1 · p. 11, Results · p. 11
• Jointly predicting situations and groundings improves both grounding and non-grounding scores, with situation context aiding localization and object localization aiding understanding.The authors also highlight JSL's ability to predict the correct noun under the GroundTruthVerb setting.Source: Results · p. 11
• JSL achieves state-of-the-art GroundTruthVerb-Value accuracy, surpassing Kernel GraphNet despite not leading in value-all.The authors suggest grounding may help JSL recover partial information and recognize more objects in complicated images where other models fail completely.Source: Results · p. 11

6 Discussion

• The discussion presents initial findings exploring future research directions enabled by Grounded Situation Recognition and SWiG.The explorations covered semantic image retrieval, conditional situation recognition, and grounded semantic chaining.Source: §6 · p. 12, Grounded Semantic aware Image Retrieval · p. 12, Conditional Grounded Situation… · p. 13

Grounded Semantic aware Image Retrieval.

• Grounded situation retrieval matches both the query’s semantics and object arrangement, while ResNet-50 and object-detection retrieval can return images with the wrong situation.The retrieved grounded situations also provide explanations, and retrieval could extend to text or mixed text-and-image queries.Source: Grounded Semantic aware Image Retrieval · p. 12, Figure 6 · p. 12

Conditional Grounded Situation Recognition.

• Conditioning a modified JSL model on a localized actor, object, or region enables queries about the situation involving that entity or location.Examples include a cellphone query invoking calling and a baby query invoking feeding; a query box can contain one or more entities.Source: Conditional Grounded Situation… · pp. 12–13, Figure 7 · p. 13

Grounded Semantic Chaining.

• Chaining CSL results across detected bounding boxes uses spatial and semantic proximity to connect local situations into higher-order image semantics.SWiG’s diverse verbs and semantic roles make these chains revealing, extending beyond the binary entity relations emphasized by scene graphs.Source: Grounded Semantic Chaining · pp. 13–14, Figure 8 · p. 13

Conclusion

• The conclusion states that GSR and SWiG enable large gains when semantic frames and groundings are predicted jointly rather than independently.It also identifies further research directions enabled by the work.Source: Conclusion · p. 14

Appendix

• The appendix provides retrieval and model implementation details, explains the changes from JSL to CSL, and adds qualitative localization and verb-prediction results.Its described comparisons include ISL versus JSL localization and visualizations for JSL’s top-five predicted verbs.Source: Appendix · p. 19

A Semantic Image Retrieval

• Grounded situation retrieval finds semantically correct images with matching viewpoints, while ResNet and object-detection methods struggle on the shown queries.The examples include a baker kneading dough and multiple hikers walking.Source: Figure 10 · p. 19, A Semantic Image Retrieval · p. 19
• The retrieval evaluation uses 1,008 query images and 24,192 search images, selecting the top five matches for each query.The query set contains two images per verb and the search set contains 48 images per verb.Source: A Semantic Image Retrieval · p. 20

A.1 ResNet-50

• ResNet-50 retrieval ranks images by the negative L2 distance between their pretrained ImageNet feature representations.Images with nearer feature representations are treated as more similar.Source: §A.1 · p. 20

A.2 Object Detections

• Object-detection similarity compares detected classes and bounding boxes, reaching its maximum when both match across images.The detections come from a modified RetinaNet, with maximum-likelihood categories and non-maximum suppression for overlapping boxes of the same category.Source: §A.2 · p. 20

A.3 Situation Recognition

• Situation similarity uses the top five predicted verbs and their associated role entities, and is nonzero only when the images share a predicted verb.It reaches 1 only when both images share the same top-ranked verb and all entities predicted for that verb match.Source: §A.3 · p. 20

A.4 Grounded Situation Recognition

• Grounded situation similarity adds predicted entity bounding boxes and increases when those boxes have greater intersection over union.The measure is described as nearly identical to situation similarity apart from the contribution of bounding-box overlap.Source: §A.4 · p. 21

B.1 RNN

• The model uses an ImageNet-pretrained ResNet-50 backbone.Source: Architecture · p. 21
• The single-layer LSTM has a hidden size of 1024 and an input size of 2816, combining 2048 image features with 512-dimensional noun and 256-dimensional verb embeddings.Source: Architecture · p. 21
• Noun embeddings are 512-dimensional, verb embeddings are 256-dimensional, and the first noun prediction uses a zero-initialized noun vector.Source: Architecture · p. 21
• The LSTM uses orthogonal weight initialization and predicts six frames, padding shorter ground-truth frames to length six.Source: Architecture · p. 21
• Training supplies the ground-truth verb embedding to the LSTM throughout, because incorrect verb predictions are also marked as incorrect noun predictions.Source: Architecture · p. 21
• The authors state that training with incorrect verb predictions offers no benefit under this evaluation rule.Source: Architecture · p. 21

Architecture

• Training uses Adam for 100 epochs, with ResNet frozen until epoch 14 and the learning rate reduced tenfold at epochs 12 and 24.The initial learning rate is 1e-4; training uses batches of 32, takes 40 hours on one 12GB TITAN V GPU, and selects the best-performing epoch's weights.Source: Training · p. 21

B.2 Object Detector

• The RetinaNet detector uses a pretrained ResNet-50, objectness scoring for 10,000 categories, and top-100 RoI-aligned boxes classified over the noun vocabulary.NMS removes overlapping low-scoring boxes; training labels boxes against ground truth at IoU ≥ 0.5, and nouns with no logit above -4 are ungrounded.Source: §B.2 · pp. 21–22
• The detector trains with batch size 64 and Adam at learning rate 1e-4, selecting epoch 26; training takes about 72 hours on eight TITAN V GPUs.Training remains slow because it performs 100 classifications per image.Source: §B.2 · p. 22

B.3 JSL

• JSL uses a single-layer LSTM with hidden size 1024, 512-dimensional noun embeddings, and 256-dimensional verb embeddings, combining its state with FPN features in a 768-channel input.For the first five epochs, it uses ground-truth boxes for local features and the previous ground-truth noun; shorter frames are padded to length six and labeled ungrounded.Source: §B.3 · pp. 22–23
• JSL trains with batch size 64 and Adam, lowering its initial learning rate of 6e-4 tenfold at epochs 10 and 20 and unfreezing the ResNet at epoch 12.It selects the best validation weights at epoch 27 and trains for about 20 hours on four TITAN RTX GPUs.Source: §B.3 · p. 23

Verb Prediction Network

• The separate verb classifier uses a ResNet-2048 feature vector and linear layer, progressively unfreezes blocks while keeping the first frozen, and trains with cross-entropy.It uses batch size 256 and takes about one hour on eight TITAN V GPUs.Source: Verb Prediction Network · p. 23

C Conditional Situation Recognition

• CSL predicts a verb from RoI-aligned features inside the queried box, then uses those features and the verb to predict the box’s role.The CSL schematic highlights its differences from JSL.Source: Figure 11 · p. 23, C Conditional Situation Recognition · p. 24
• At the predicted role’s LSTM pass, CSL replaces the model’s predicted box with the input box; at other passes, it uses predicted boxes.The network checks each iteration against the queried box’s predicted role.Source: C Conditional Situation Recognition · p. 24

D Qualitative

• JSL localizes entities more accurately than ISL when multiple same-category objects make role selection difficult or when precise bounding boxes are needed.The examples show ISL correctly classifying nouns but choosing the wrong entity or producing an inaccurate box; sequential prediction may improve localization.Source: D Qualitative · p. 24, Figure 12 · p. 25
• For each image, the model generates grounded situations for its top 5 probable verbs, with examples showing interactions between participants and their shared actions.One example describes two girls studying, including that one is explaining something to the other.Source: D Qualitative · p. 24, Figure 13 · p. 26
• The correct verb appears in the top 5 in some examples but is missed when actions are unusual or occur in strange contexts.Purposefully spilling water on a keyboard is one example of an action the model misses in its top-five predictions.Source: D Qualitative · p. 25, Figure 13 · p. 26