PaperlayerView exact revision on arXiv

Source-linked AI summary

Actions as Moving Points

Yixuan Li, Zixu Wang, Limin Wang, Gangshan Wu

arXiv:2001.04608v3cs.CV

TL;DR

Anchor-based action tubelet detectors face computational and localization challenges as video duration increases. MOC represents actions as moving-point trajectories and estimates their centers, movement, and box sizes; on JHMDB and UCF101-24, it achieves state-of-the-art frame and video mAP at reasonable computational cost.

  • Problem

    Anchor-based tubelet detectors rely on heuristic box designs and dense placements, creating computational costs and challenges for precise localization as clip duration increases.

    Source: §1 · p. 2

  • Method

    MOC represents each action tubelet by a key-frame center, offsets for other frames, and directly regressed box sizes, using three cooperating branches before linking tubelets into longer tubes.

    Source: §1 · pp. 2–3, §3.4 · p. 8

  • Results

    On two challenging datasets, MOC achieves state-of-the-art performance in both frame mAP and video mAP while maintaining reasonable computational cost.

    Source: §5 · p. 15

  • Takeaways & Limitations

    The authors attribute MOC’s superior performance largely to the cooperative modeling ability of its three branches.

    Source: §5 · p. 15

    The Box Branch assumes box detection depends only on the current frame, not temporal information.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

The existing action tubelet detectors often depend on heuristic anchor design and placement, which might be computationally expensive and sub-optimal for precise localization. In this paper, we present a conceptually simple, computationally efficient, and more precise action tubelet detection framework, termed as MovingCenter Detector (MOC-detector), by treating an action instance as a trajectory of moving points. Based on the insight that movement information could simplify and assist action tubelet detection, our MOC-detector is composed of three crucial head branches: (1) Center Branch for instance center detection and action recognition, (2) Movement Branch for movement estimation at adjacent frames to form trajectories of moving points, (3) Box Branch for spatial extent detection by directly regressing bounding box size at each estimated center. These three branches work together to generate the tubelet detection results, which could be further linked to yield video-level tubes with a matching strategy. Our MOC-detector outperforms the existing state-of-the-art methods for both metrics of frame-mAP and video-mAP on the JHMDB and UCF101-24 datasets. The performance gap is more evident for higher video IoU, demonstrating that our MOC-detector is particularly effective for more precise action detection. We provide the code at https://github.com/MCG-NJU/MOC-Detector.

1 Introduction

The introduction motivates tubelet detection as a way to use short-term temporal information, while identifying limitations in frame-wise detection and anchor-based designs. It presents MOC-detector, which represents an action tubelet through a key-frame center, offsets, and bounding-box sizes, and reports strong benchmark results.

1 Introduction

  • Frame-wise action detectors link or track detections into tubes, but the introduction says they fail to capture temporal information well during frame-level detection.Clip-level methods instead take a frame sequence and directly output tubelets, or short sequences of bounding boxes.Source: §1 · pp. 1–2
  • Anchor-based tubelet detectors rely on many predefined boxes, making them sensitive to design choices and less efficient as clip duration increases.Extending 2D anchors into temporal cuboids also limits flexibility in modeling adjacent-frame box variation.Source: §1 · p. 2
  • MOC-detector represents a tubelet with a key-frame center and offsets for other frames, then regresses bounding-box size at each moving point.Its Center, Movement, and Box branches detect the center and category, estimate offsets, and predict box sizes, respectively.Source: §1 · p. 3
  • MOC-detector outperforms existing state-of-the-art approaches on frame-mAP and video-mAP across UCF101-24 and JHMDB, particularly at higher IoU criteria, with detection efficiency around 25 FPS.The reported comparison covers both datasets and both metrics.Source: §1 · p. 3

2 Related Work

Related work contrasts anchor-based and anchor-free object detectors, then reviews frame-level and clip-level approaches to spatio-temporal action detection. The paper positions its method as an anchor-free tubelet detector based on moving points.

2.1 Object Detection

  • Anchor-based detectors use predefined boxes, while anchor-free methods can be competitive without computation-intensive anchors and region-based CNNs.Examples include CornerNet's paired corners and CenterNet's center-point representation.Source: §2.1 · p. 3

2.2 Spatio-temporal Action Detection

  • Frame-level action detectors link or track per-frame detections into tubes, but do not fully use video temporal information.The reviewed approaches include linking detections and tracking by detection.Source: §2.2 · pp. 3–4
  • Clip-level methods model temporal information through tubelets, proposal linking, or progressive refinement; ACT regresses tubelets from anchor cuboids.STEP progressively refines proposals to address large displacement and uses longer temporal information.Source: §2.2 · p. 4
  • The paper proposes an anchor-free tubelet detector that treats each action instance as a trajectory of moving points, reporting effectiveness especially at high video IoU.The related-work discussion contrasts this with anchor-based approaches that can depend on anchor design and large numbers of boxes.Source: §2.2 · p. 4

3 Approach

MOC-detector represents an action tubelet as a trajectory of moving points, using three branches to detect its center and category, estimate movement, and regress box size. Tubelets from short clips are linked into action tubes across longer videos.

3 Approach

  • A 2D backbone extracts frame-level features for three anchor-free branches, whose tubelet detections are linked across time into video-level action tubes.The branches localize and classify the key-frame center, estimate movement across frames, and regress bounding-box size at each center.Source: §3 · pp. 4–5

3.1 Center Branch: Detect Center at Key Frame

  • The Center Branch uses multi-frame features to detect the action center and recognize its category on the key frame.It estimates a class-specific center heatmap in a fully convolutional manner.Source: §3.1 · pp. 5–6

3.2 Movement Branch: Move Center Temporally

  • The Movement Branch predicts center offsets from the key frame to other frames, forming a trajectory from each detected center.At inference, the predicted offsets are added to the key-frame center to produce moving points across the clip.Source: §3.2 · pp. 6–7

3.3 Box Branch: Determine Spatial Extent

  • The Box Branch estimates bounding-box width and height at each trajectory point, assuming box detection depends only on the current frame.It uses single-frame features and is shared across the clip’s frames.Source: §3.3 · pp. 7–8

3.4 Tubelet Linking

  • The linking procedure connects clip-level tubelets into video-level tubes, averaging tubelet scores and box coordinates within each link.A link stops when it is not extended for K consecutive frames; low-confidence and short-duration tubes are discarded.Source: §3.4 · p. 8

4 Experiments

The experiments test MOC on two action-detection datasets, examine design and temporal-length choices, and compare its accuracy and speed with existing methods. MOC outperforms existing frame-level and tubelet detectors on both datasets, with a larger performance gap at high video IoU.

4.1 Experimental Setup

  • Experiments use UCF101-24 split one and JHMDB results averaged over three splits, evaluating detection with frame mAP and video mAP.Source: §4.1 · pp. 8–9

4.2 Ablation Studies

  • 1.56% for frame mAP@0.5 and 11.05% for video mAP@0.5 separate No Movement from Semi Movement; Full Movement performs slightly better than Semi Movement on both metrics.Movement has a relatively small effect on frame mAP but contributes much to video mAP, where linking accumulates movement errors.Source: Effectiveness of Movement Branch · pp. 9–10, Table 1 · p. 10
  • Center Movement performs best among the tested movement designs and is adopted in MOC; Flow Guided, Cost Volume, and Accumulated Movement have reported accuracy or efficiency drawbacks.The authors attribute drawbacks to inaccurate pixel motion, accumulated error, harder convergence, and training-inference inconsistency.Source: Study on Movement Branch Design · pp. 10–11, Table 2 · p. 11
  • Detection performance rises as K increases from 1 to 7, with smaller gains from K = 5 to K = 7, then drops at K = 9; MOC uses K = 7.The authors attribute the drop to the greater difficulty of predicting movement for longer inputs.Source: Table 3 · p. 11, Study on Input Sequence Duration · pp. 11–12

4.3 Comparison with the State of the Art

  • MOC outperforms frame-level and tubelet detectors on both datasets across frame-mAP and video-mAP, with larger gains at high video IoU; it is comparable to 3D-backbone methods.Source: Table 4 · p. 12, §4.3 · pp. 12–13

4.4 Runtime Analysis

4.5 Visualization

  • Tubelet detections show fewer missed actors and more accurate localization than per-frame detections, while temporal context can reduce classification errors.The visualization compares K = 1 with K = 7; its detection threshold is 0.4.Source: Figure 6 · p. 14, §4.5 · pp. 14–15

5 Conclusion and Future Work

MOC treats each action instance as a trajectory of moving points and directly regresses box sizes at estimated centers. On two challenging datasets, it achieves state-of-the-art frame mAP and video mAP at reasonable computational cost.

5 Conclusion and Future Work

  • MOC achieves state-of-the-art frame mAP and video mAP on two challenging datasets while maintaining reasonable computational cost.The paper attributes the performance largely to the cooperative modeling ability of its three branches.Source: §5 · p. 15
  • The detector represents each action instance as a moving-point trajectory and directly regresses bounding-box sizes at estimated centers across frames.Source: §5 · p. 15
  • Future work will extend MOC to longer-term modeling and action-boundary modeling in the temporal dimension.Source: §5 · p. 15

Appendix A: Study on Hyper-parameters

The hyperparameter studies examine the Center Branch candidate cap and the loss weights. The reported settings are N=100 and a=1, b=0.1, with only slight changes after N=40.

Appendix A: Study on Hyper-parameters

  • At inference, the Center Branch retains the top N instances across categories after max pooling.Source: Appendix A · pp. 15–16
  • N=100 is the Center Branch inference cap after max pooling, and the detection result changes only slightly once N exceeds 40.Figure 7 reports FrameAP@0.5 on UCF101-24 with K=5 and RGB input.Source: Appendix A · pp. 15–16, Figure 7 · p. 15
  • The linear search reports a=1 and b=0.1 as the best loss weights.Figure 8 reports FrameAP@0.5 on UCF101-24 with K=5 and RGB input.Source: Appendix A · p. 16, Figure 8 · p. 16

Appendix B: More exploration on Box Branch

The Box Branch study compares estimating bounding boxes separately for each frame with stacking features to add temporal information. The reported result favors separate, single-frame box estimation.

Appendix B: More exploration on Box Branch

  • The study compares union, which stacks features to add temporal information, with separate MOC estimation for each frame.The comparison uses RGB input and K=5.Source: Table 5 · p. 16
  • Adding temporal information to box estimation by stacking features reduces performance, indicating a single frame is sufficient for bounding-box detection.Source: Appendix B · p. 16
  • The Box Branch design estimates bounding boxes separately rather than incorporating temporal features into bbox estimation.The passage identifies the separate approach as MOC.Source: Appendix B · p. 16, Table 5 · p. 16

Appendix C: Error Analysis

The error analysis finds different dominant errors on the two datasets: time error on UCF101-24 and classification error on JHMDB. With two-stream fusion on UCF101-24, fusion improves performance except for time error.

Appendix C: Error Analysis

  • The analysis distinguishes classification, localization, time, missed-detection, and other errors using the stated IoU and temporal-extent criteria.Source: Appendix C · p. 17
  • 10.18% time error is the major error on UCF101-24, while 25.43% classification error is the major error on JHMDB split 1.Figure 9 reports the analysis with K=7 and two-stream fusion; the green part represents correct detections.Source: Appendix C · p. 17, Figure 9 · p. 17
  • On UCF101-24, the spatial stream performs better than the temporal stream for classification error and missed detection, attributed to its richer information.The comparison uses K=7 and two-stream error analysis.Source: Appendix C · p. 17, Figure 10 · p. 18
  • Two-stream fusion improves performance except for time error, indicating harm to temporal localization on UCF101-24.Source: Appendix C · p. 18, Figure 10 · p. 18

Appendix D: More Results on JHMDB

The JHMDB comparison reports MOC outperforming Gu et al. across metrics and performing comparably to Sun et al., with lower estimated computation. The table evaluates tubelets of length K=7 with two-stream fusion.

Appendix D: More Results on JHMDB

  • More than 3× computation savings versus Gu et al. and more than 2× versus Sun et al. accompany MOC’s reported JHMDB comparison.MOC outperforms Gu et al. across metrics and performs comparably to Sun et al.; the computation estimates are approximate because neither baseline provides implementation code.Source: Table 6 · p. 18, Appendix D · p. 19
  • Table 6 compares methods on JHMDB using tubelet length K=7 and two-stream fusion, and distinguishes MOC pretraining on ImageNet, COCO, or UCF101-24.The UCF101-24-pretrained MOC is pretrained for action detection.Source: Table 6 · p. 18
  • Two-stage detectors with 3D backbones perform comparably to MOC on UCF101-24 but better than MOC on JHMDB.The comparison is described in Section 4.3.Source: Appendix D · pp. 18–19

Related papers

Loading 2001.04608v3…