PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Semantic Role Labeling

Saurabh Gupta, Jitendra Malik

arXiv:1505.04474v1cs.CV

TL;DR

The paper addresses the gap between coarse action recognition and detailed visual understanding of who is acting and which objects fill the action’s semantic roles. It introduces Visual Semantic Role Labeling, builds a COCO-based dataset, and evaluates baseline detectors, finding the task challenging with performance varying by action and object characteristics.

  • Problem

    Action recognition often classifies an image or video or localizes the acting person without associating scene objects with the action’s semantic roles.

  • Method

    The paper annotates COCO with 16K people instances in 10K images, 26 action classes, semantic-role objects, and CNN-based baseline algorithms.

  • Results

    Baseline performance varies by action and object characteristics, with agent detection reaching 57.5% mAP and role detection reaching 7.9% mAP using regression, versus 4.5% for the mean-location baseline.

  • Takeaways & Limitations

    The dataset and tasks establish a benchmark for detailed action understanding that jointly detects people, classifies actions, and localizes semantically related objects.

Abstract

from arXiv · show

In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.

1. Introduction

Visual Semantic Role Labeling extends action recognition from coarse activity classification to detecting agents, fine-grained actions, and objects assigned to semantic roles. The paper introduces this task and an annotated COCO-based dataset with baseline methods.

  • Existing action-recognition systems typically classify activities or localize the person, but do not provide a complete fine-grained scene understanding.
  • The proposed output detects fine-grained actions and localizes their agents, instruments, objects, and other semantic roles.
  • Associating objects with actions supports reasoning about scene state, future events, commonsense, and broader activities.
  • Visual Semantic Role Labeling grounds verb roles from natural-language semantic role labeling in images.
  • Examples include people performing multiple actions simultaneously, such as drinking while sitting or sitting while riding.

2. Related Work

Prior work studied action classification, agent localization, pose, object context, video activities, and image captioning. These approaches generally lacked joint annotations and evaluation connecting actions with their interacting objects and semantic roles.

  • Related work spans static action classification, video activity analysis, pose, object context, and image captioning.
  • PASCAL VOC and MPII support action or pose analysis, but neither annotates the objects of interaction emphasized by this work.
  • Sports, PPMI, and TBH studies model human pose and object context, yet do not quantify the joint action-and-object task.
  • Video datasets commonly target full-video action classification or detection of the agent performing an action.
  • Image-captioning systems may localize words but often capture only the most salient action rather than detailed action-role relationships.

3. V-COCO Dataset

V-COCO is constructed by selecting action-relevant COCO images, annotating salient people and their actions, and associating objects with semantic roles through AMT-based procedures. The resulting dataset supports fine-grained action-understanding tasks with inherited COCO annotations and defined train, validation, and test splits.

  • Dataset construction: V-COCO builds on COCO to provide challenging scenes, extensive object annotations, segmentation masks, captions, and potential enrichment from future COCO annotations.The dataset uses COCO because it contains 160K images, 80 object classes, segmentation masks, and five captions per image.
  • Dataset construction: The annotation pipeline identifies person-subject verbs from COCO captions, manually selects basic actions, and removes visually ambiguous verbs.The authors initially select 30 verbs and drop pick, place, give, and take because they are ambiguous in single images.
  • Dataset construction: Images are ranked using caption-based person-action and object-association scores, with independently selected pools merged into a common image set using an integer program.The procedure considers the top 8000 images per verb before obtaining person annotations and merging action-specific positives.
  • Annotation procedure: The dataset annotates salient people with binary labels for each action, using five AMT workers per person per action, then obtains YES/NO labels for objects in semantic roles.Role questions ask whether the highlighted person is interacting with a highlighted object in the specified way.
  • Dataset statistics: The authors construct train, validation, and test splits, placing COCO validation images in V-COCO test and splitting the remaining images into train and validation.A joint COCO training-and-validation pool is annotated before the splits are created.
  • Annotation procedure: AMT tasks use simple binary interfaces with fixed actions, test questions, worker filtering, and additional object highlighting for role annotation.HITs contain 450 questions, including 50 test questions; submission accuracy below 90% is blocked.
  • Dataset statistics: V-COCO contains 10,346 images and 16,199 people instances, each with binary labels for 26 actions and associated semantic-role annotations.The dataset also inherits COCO annotations, including bounding boxes, and includes annotations for non-salient people and crowd regions for detection studies.
  • Tasks and metrics: The annotations enable previously unstudied fine-grained action-understanding tasks involving actions, people, and objects in semantic roles.The dataset is designed to support tasks beyond conventional action labels or person localization.

4. Methods

The baselines detect people and classify their actions, then localize semantic-role objects either by regression or by combining object detection with spatial consistency.

  • Object detection: The baseline pipeline begins by training R-CNN object detectors for 80 COCO categories with a VGG network fine-tuned using Fast R-CNN.Training uses 77K COCO images while holding out the V-COCO train and validation images.
  • Agent detection: The agent detector first detects people and classifies each detected person across actions as a multi-label problem.Model A is trained on person proposals with intersection over union above 0.5 and fine-tunes VGG features.
  • Agent detection: At test time, action probabilities are multiplied by person-detector probabilities to produce final action scores.
  • Role localization: Model B regresses each semantic-role box in the detected agent’s coordinate frame using target-agent centers and box dimensions.The regression targets encode relative location and scale between the role object and person boxes.
  • Role localization: The detector-based approach selects the highest-scoring relevant object box for each detected agent and action.Its score combines object-category detection probability with a learned Gaussian distribution over agent-object deformations.

5. Experiments

Experiments evaluate proposal coverage, agent detection, role detection, and error modes on V-COCO. The strongest baselines benefit from fine-tuning and spatial modeling, but action classification and difficult object localization remain major challenges.

  • Evaluation: The V-COCO evaluation reports proposal recall, agent-detection AP, and role-detection AP for four VGG-based baselines.Results are reported on the V-COCO validation set, with object proposal coverage measured by intersection over union.
  • Agent detection: 62.54% average precision is obtained by the VGG Fast R-CNN person detector on the V-COCO validation set.
  • Agent detection: 57.5% mean average precision is achieved for agent detection, while an SVM on VGG fc7 features reaches 46.8%.Performance is higher for actions with distinctive scenes or objects and lower for difficult object, spatial, or pose reasoning.
  • Role detection: 26.4% mean average precision is achieved by model C for role detection, outperforming model C0 at 23.4%, model B at 7.9%, and model B0 at 4.5%.Model C adds spatial consistency to relevant object-detection scores.
  • Error analysis: Incorrect action classification is the dominant error, alongside object mislocalization, person mislocalization, mis-pairing, and background hallucination.Mislocalization is especially frequent for unusual poses or small, nondistinctive, rapidly moving objects.
Loading 1505.04474v1…