PaperlayerView exact revision on arXiv

Source-linked AI summary

Spot On: Action Localization from Pointly-Supervised Proposals

Pascal Mettes, Jan C. van Gemert, Cees G. M. Snoek

arXiv:1604.07602v2cs.CV

TL;DR

Action localization often depends on costly, carefully annotated boxes. This paper uses sparse points and action proposals in a Multiple Instance Learning approach to mine training examples. On UCF Sports and UCF 101, point annotations yield results comparable to box supervision while being significantly faster to annotate.

  • Problem

    Action-localization classifiers commonly rely on carefully annotated bounding boxes, which are cumbersome, tedious, and error-prone to produce.

    Source: §1 · p. 1

  • Method

    The approach scores spatio-temporal proposals by their overlap with points annotated on a sparse subset of frames and incorporates the scores into a Multiple Instance Learning objective.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    On UCF Sports and UCF 101, point annotations yield results comparable to box supervision while being significantly faster to annotate, and proposal training retains localization performance.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    The method provides a way to annotate action locations with points on a fraction of frames and supplements Hollywood2 with location annotations as Hollywood2Tubes.

    Source: §1 · p. 3, §6 · p. 14

    Hollywood2Tubes includes difficult localization cases absent from current datasets, including cinematographic effects and camera switches.

    Source: §5.3 · p. 12

Abstract

from arXiv · show

We strive for spatio-temporal localization of actions in videos. The state-of-the-art relies on action proposals at test time and selects the best one with a classifier trained on carefully annotated box annotations. Annotating action boxes in video is cumbersome, tedious, and error prone. Rather than annotating boxes, we propose to annotate actions in video with points on a sparse subset of frames only. We introduce an overlap measure between action proposals and points and incorporate them all into the objective of a non-convex Multiple Instance Learning optimization. Experimental evaluation on the UCF Sports and UCF 101 datasets shows that (i) spatio-temporal proposals can be used to train classifiers while retaining the localization performance, (ii) point annotations yield results comparable to box annotations while being significantly faster to annotate, (iii) with a minimum amount of supervision our approach is competitive to the state-of-the-art. Finally, we introduce spatio-temporal action annotations on the train and test videos of Hollywood2, resulting in Hollywood2Tubes, available at http://tinyurl.com/hollywood2tubes.

1 Introduction

Action localization commonly depends on carefully annotated boxes, which are costly to produce. This paper uses spatio-temporal proposals and sparse point supervision to mine training examples, aiming to retain localization accuracy with less annotation effort.

1 Introduction

  • Action-localization training typically assumes carefully annotated boxes, while this work uses proposals with minimum supervision to reduce annotation effort.The authors note that action-localization MIL without added supervision yields poor accuracy in their experiments.Source: §1 · pp. 1–2
  • The approach trains action-localization classifiers on spatio-temporal proposals instead of ground-truth tubes without decreasing localization accuracy.The proposal-mining method uses point supervision to select proposals with a good spatio-temporal fit to the action.Source: §1 · pp. 2–3
  • The method adds an overlap measure between proposals and points to a Multiple Instance Learning objective for mining action proposals.The approach starts with point annotations on a sparse subset of frames and aims to identify the single proposal that best represents the action.Source: Figure 1 · p. 2, §1 · p. 3
  • The authors supplement the Hollywood2 dataset with action-location annotations, creating Hollywood2Tubes.They also report experiments on Hollywood2Tubes alongside UCF Sports and UCF 101.Source: §1 · p. 3

2 Related work

Related work includes proposal-based action localization, weakly supervised methods, and efforts to reduce annotation effort. The paper positions sparse points as a middle ground between class-label-only supervision and costly bounding-box annotation.

2 Related work

  • Action proposals can provide high recall without supervision, while person detectors may miss contextual actions such as driving or interactions such as kissing.The cited proposal methods use supervoxels or trajectory clustering.Source: §2 · p. 3
  • The paper identifies spatio-temporal annotation effort, rather than the number of available videos, as a barrier to scaling action-localization datasets.It notes that millions of action videos and hundreds of classes are available, while many annotated datasets contain far fewer clips.Source: §2 · p. 3
  • Unlike tools that streamline box or polygon drawing, point annotations can be added to existing tools and avoid algorithmic bias in selecting what to annotate.The authors contrast their approach with active learning and trained detectors that can bias annotation toward a recognition method.Source: §2 · p. 3
  • For action localization, the paper proposes points as a middle ground because using no labels or only class labels performs below fully supervised results.Compared with full boxes, points greatly reduce annotation time while retaining accuracy.Source: §2 · p. 4

3 Strong action localization using cheap annotations

The method uses action proposals as candidate training examples and point annotations to help mine those that best fit the action. It combines point–proposal overlap with Multiple Instance Learning, then alternates between proposal mining and classifier training.

3 Strong action localization using cheap annotations

  • The approach starts from the hypothesis that a proposal can substitute for a ground-truth tube in training and mines the best candidate from many proposals with minimal supervision.Proposal algorithms provide hundreds to thousands of candidates per video.Source: §3 · p. 4

3.1 Cheap annotations: action class labels and pointly-supervision

  • Each video is a MIL bag of proposals, and sparse frame-level points supplement its global action label to help distinguish proposals that overlap annotated points.Points exclude proposals with no overlap, but do not uniquely identify a proposal, motivating an overlap measure.Source: §3.1 · p. 4

3.2 Measuring overlap between points and proposals

  • The overlap measure rewards points near proposal-box centers, averages their scores, and penalizes large boxes relative to the video frame.The center-distance score is normalized by the distance to the furthest box side; the size penalty addresses a center bias in large proposals.Source: §3.2 · p. 5

3.3 Mining proposals overlapping with points

  • Proposal mining inserts the overlap score into a MIL-SVM objective and alternates between mining proposals and training the classifier before fitting a final SVM.The objective is non-convex because it jointly optimizes classifier parameters and mined proposals; mining uses the classifier as a likelihood and overlap as a prior.Source: §3.3 · pp. 5–6

4 Experimental setup

The experiments use UCF Sports and UCF 101, both with bounding-box annotations for training and test videos. Implementation uses APT proposals and fixed-size Fisher Vector representations, then evaluates ranked proposal detections with AP and, on UCF Sports, AUC.

4.1 Datasets

  • UCF Sports has 150 videos across 10 categories, while the annotated UCF 101 subset has 3,204 videos across 24 categories.The UCF 101 split uses 2,290 training and 914 test videos; both datasets have box annotations for training and test videos.Source: §4.1 · p. 6

4.2 Implementation details

  • APT proposals performed slightly better than Tubelets, so all reported results use APT with 54,656-dimensional Fisher Vector representations.Improved Dense Trajectories features are PCA-reduced to 128 dimensions and aggregated using a 128-cluster codebook.Source: §4.2 · p. 7
  • Training uses 10 optimization iterations, three random video splits, 100 negative proposals per other-action video, and SVM regularization λ=100.Source: §4.2 · p. 7
  • A top proposal is a positive detection when its video-level IoU reaches threshold t; results use AP, with AUC also reported for UCF Sports.IoU averages framewise box overlap over frames where either the proposal or annotation is present.Source: §4.2 · p. 7

5 Results

Across UCF Sports and UCF 101, proposal-trained classifiers and point-guided proposal mining retain localization performance close to box-supervised training, while sparse point annotations reduce effort. The experiments also introduce Hollywood2Tubes and compare point-supervised results with existing methods.

5.1 Training without ground truth tubes

  • Across both datasets and thresholds, best-proposal training is similar to ground-truth-tube training, and point-mined proposals are competitive with ground-truth-trained localization.Best-proposal average IoU is 0.642 on UCF Sports and 0.400 on UCF 101; imperfect fits were not detrimental to results.Source: §5.1 · pp. 7–9, Figure 2 · p. 8
  • On UCF 101, temporal overlap averages 0.74 versus 0.38 spatial overlap, and 37.2% of actions are absent from top selected proposals at IoU 0.2.Top-ranked proposals average 2.67 times the size of the ground truth; 26.7% of the top ten proposals are below IoU 0.2.Source: §5.1 · pp. 8–9

5.2 Must go faster: lowering the annotation frame-rate

  • At a 10-frame annotation rate, localization performance is generally retained with 45x or greater speedup; points remain competitive with boxes at 10x to 80x speedup.The authors recommend annotating at least 10% of frames with points.Source: Figure 4 · p. 10, §5.2 · p. 10
  • Iterative point-guided mining improves final proposal IoU in two examples, from 0.194 to 0.627 and from 0.401 to 0.526, but reverts to the initial best proposal in another.In that third example, a better proposal was mined midway through the iterations.Source: §5.2 · p. 10, Figure 5 · p. 11

5.3 Hollywood2Tubes: Action localization for Hollywood2

  • Hollywood2Tubes covers 12 actions across 823 training and 884 test videos, with point annotations for training and bounding boxes for test evaluation.The training set contains 1,026 action instances; the test videos are annotated every 10 frames with boxes.Source: §5.3 · p. 11
  • Hollywood2Tubes achieves a MABO of 0.47, but recall and Average Precision do not always align, with the results indicating room for improvement.Sitting up has above-average recall but the second-lowest Average Precision; Fighting a Person is among the top Average Precision performers.Source: Figure 6 · p. 12, §5.3 · p. 12
  • Hollywood2Tubes includes difficult cases involving multiple people, context-defined actions, and co-occurring actions, as well as cinematographic effects and camera changes.These scenarios are described as adding to the dataset’s localization complexity.Source: §5.3 · p. 12, Figure 7 · p. 13

5.4 Comparison to the state-of-the-art

  • At 0.2 overlap, UCF Sports AUC is 0.545 versus APT’s 0.546 and near the state-of-the-art 0.559; UCF 101 trails Weinzaepfel et al., while points beat global labels on Hollywood2Tubes.The table compares localization results across the three datasets.Source: §5.4 · p. 13, Table 1 · p. 14

6 Conclusions

The paper concludes that action localization can use training proposals and sparse point annotations instead of precise per-frame boxes. It also introduces Hollywood2Tubes with action-location annotations.

6 Conclusions

  • Proposals can train action classifiers without reducing localization performance relative to using ground-truth tubes.Source: §6 · p. 14
  • Point supervision achieves localization performance comparable to full box supervision while being significantly faster to annotate.Source: §6 · p. 14
  • Hollywood2Tubes supplements the Hollywood2 dataset with action-location annotations for train videos.Source: §6 · p. 14

Supplementary materials

The supplementary materials cover the Hollywood2Tubes annotation protocol, annotation statistics, and visualizations of box annotations by action.

Supplementary materials

Annotation protocol

Hollywood2Tubes uses boxes to specify action locations and points to mark box centers. Its protocol adapts the annotated region to the action, and the aggregated annotations are generally center-oriented.

Annotation protocol

  • For point annotations, only the center of the corresponding box is marked, rather than the complete box.Source: Annotation protocol · p. 17
  • AnswerPhone boxes cover the answering person’s head and phone-holding hand; DriveCar boxes cover the driver and upper steering wheel, or the whole distant car.Source: Annotation protocol · p. 17
  • Eat boxes unite the people eating, while FightPerson boxes include both fighters or, in a chaotic brawl, the fighting group.Source: Annotation protocol · p. 17
  • Handshake boxes cover the participants’ arms, HugPerson boxes their heads and upper torsos, and Kiss boxes their heads.Source: Annotation protocol · p. 17
  • Run boxes cover the running person; SitDown and SitUp boxes cover the complete person over the specified movement transitions.Source: Annotation protocol · p. 17
  • Aggregated point and box annotations are center-oriented, with box annotations showing a rule-of-thirds bias; the annotation-statistics table notes that actions are untrimmed.Source: Figure 8 · p. 18, Table 2 · p. 18

Annotation statistics

The annotation aggregations are center-oriented, while box annotations show a rule-of-thirds bias. The reported statistics reflect that Hollywood2 actions are not trimmed.

Annotation statistics

Annotation examples

Figure 9 shows example box annotations from Hollywood2Tubes test videos, with one example frame for each of 12 actions illustrating the videos’ diversity and complexity.

Annotation examples

  • Figure 9 presents example box annotations for Hollywood2Tubes test videos, showing one frame for each of 12 actions.The examples illustrate the diversity and complexity of the videos used for action localization.Source: Annotation examples · p. 18, Figure 9 · p. 19
Loading 1604.07602v2…