PaperlayerView exact revision on arXiv

Source-linked AI summary

BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation

Haisheng Su, Weihao Gan, Wei Wu, Yu Qiao, Junjie Yan

arXiv:2009.07641v5cs.CV

TL;DR

Temporal proposal methods face noisy boundaries, weak confidence estimates, and unmodeled proposal relations and sampling imbalance. BSN++ combines context-aware complementary boundary prediction, proposal-relation modeling, and scale-balanced resampling; experiments report effectiveness for proposal generation and detection on ActivityNet-1.3 and THUMOS14.

  • Problem

    Boundary noise, inferior proposal-confidence quality, unmodeled proposal relations, and imbalanced proposal distributions are challenges in temporal proposal generation.

    Source: Introduction · p. 2

  • Method

    BSN++ jointly generates boundary and confidence maps using a complementary context-aware boundary generator, a two-attention proposal relation block, and two-stage resampling.

    Source: Introduction · p. 2

  • Results

    Experiments on ActivityNet-1.3 and THUMOS14 demonstrate effectiveness in proposal generation and detection, and combining BSN++ with existing classifiers achieves state-of-the-art temporal action detection performance.

    Source: Introduction · p. 2, Conclusion · pp. 7–8

  • Takeaways & Limitations

    BSN++ generates boundary and confidence maps simultaneously in a unified network.

    Source: Introduction · p. 2, Conclusion · p. 7

Abstract

from arXiv · show

Generating human action proposals in untrimmed videos is an important yet challenging task with wide applications. Current methods often suffer from the noisy boundary locations and the inferior quality of confidence scores used for proposal retrieving. In this paper, we present BSN++, a new framework which exploits complementary boundary regressor and relation modeling for temporal proposal generation. First, we propose a novel boundary regressor based on the complementary characteristics of both starting and ending boundary classifiers. Specifically, we utilize the U-shaped architecture with nested skip connections to capture rich contexts and introduce bi-directional boundary matching mechanism to improve boundary precision. Second, to account for the proposal-proposal relations ignored in previous methods, we devise a proposal relation block to which includes two self-attention modules from the aspects of position and channel. Furthermore, we find that there inevitably exists data imbalanced problems in the positive/negative proposals and temporal durations, which harm the model performance on tail distributions. To relieve this issue, we introduce the scale-balanced re-sampling strategy. Extensive experiments are conducted on two popular benchmarks: ActivityNet-1.3 and THUMOS14, which demonstrate that BSN++ achieves the state-of-the-art performance. Not surprisingly, the proposed BSN++ ranked 1st place in the CVPR19 - ActivityNet challenge leaderboard on temporal action localization task.

Introduction

Temporal proposals must cover action instances with accurate boundaries and provide reliable confidence for retrieval. BSN++ addresses boundary context, proposal relations, and imbalanced sampling, and reports state-of-the-art temporal action detection performance when combined with existing classifiers.

Introduction

  • High-quality temporal proposals need to cover action instances and provide accurate confidence scores for later retrieval.Temporal coverage includes both high recall and temporal overlap.Source: Introduction · p. 1
  • BSN++ uses a U-shaped boundary regressor with nested skip connections and bidirectional boundary matching to exploit temporal context and improve boundary precision without extra parameters.The starting classifier predicts ending locations on reversed video, and vice versa.Source: Introduction · p. 2
  • Its proposal relation block models proposal relations with two self-attention modules, while two-stage resampling balances positive-negative samples and proposal scales.The resampling combines IoU-balanced sampling with scale-balanced resampling.Source: Introduction · p. 2
  • BSN++ combined with existing action classifiers achieves state-of-the-art temporal action detection performance.The paper reports experiments to reveal the method's effectiveness.Source: Introduction · p. 2

Related Work

The related work covers visual feature extraction, imbalanced-data training, and temporal proposal methods. BSN++ is positioned as addressing limitations in boundary prediction, proposal-relation modeling, and sampling balance.

Action Recognition

  • Action-recognition features are extracted with two-stream networks for appearance and motion or 3D networks for spatial and temporal information.These networks provide visual feature sequences for temporal proposal and detection tasks.Source: Action Recognition · p. 2

Imbalanced Distribution Training

  • Imbalanced-data methods include re-sampling, reweighting, and detector-specific approaches; BSN++ applies scale-balanced resampling on top of IoU-balanced sampling.Its motivation is the imbalanced mini-batch loss distribution across proposal durations.Source: Imbalanced Distribution Training · p. 2

Temporal Action Detection and Proposals

  • Prior temporal proposal methods use top-down windows or anchors, or bottom-up boundary and actionness scores; BSN++ adds rich-context boundary prediction, proposal-relation modeling, and two-stage resampling.The related-work discussion notes noisy probability sequences, unmodeled proposal relations, and remaining proposal imbalance.Source: Temporal Action Detection and Proposals · pp. 2–3

Our Approach

BSN++ generates dense temporal proposals through boundary prediction and proposal-confidence modeling, with sampling designed to balance positive/negative proposals and proposal durations. Its boundary generator uses complementary predictions, while the relation block models proposal dependencies.

Problem Definition

  • The task is to produce temporal intervals that cover annotated action instances with high recall and temporal overlap.Training uses annotated instances; testing predicts a proposal set.Source: Problem Definition · p. 3

Video Feature Encoding

  • Video features are extracted at regular intervals, concatenating the two-stream network outputs into a sequence for BSN++.The paper uses spatial and temporal streams for feature encoding.Source: Video Feature Encoding · p. 3

Proposed Network Architecture: BSN++

  • BSN++ uses a unified network with a Base Module, Complementary Boundary Generator, and Proposal Relation Block to construct boundary and confidence maps.The Base Module models temporal information and shares its output features with the two following modules.Source: Proposed Network Architecture: BSN++ · p. 3
  • The boundary generator uses a U-shaped encoder-decoder to capture global context and local detail, with nested skip connections bridging encoder and decoder features.It also uses deep supervision and down-/up-sampling layers to expand and recover temporal resolution.Source: Proposed Network Architecture: BSN++ · pp. 3–4
  • Complementary forward and reversed predictions swap starting and ending boundary roles, then fuse their heatmaps to construct a dense boundary map.The two directions are optimized in parallel with a consistency constraint on intermediate features.Source: Proposed Network Architecture: BSN++ · p. 4
  • The proposal relation block models position dependencies and channel inter-dependencies, then combines branch outputs for pixel-level confidence prediction.Position-aware attention targets global dependencies; channel-aware attention enhances proposal features through channel-wise matrix multiplication.Source: Figure 4 · p. 4, Proposed Network Architecture: BSN++ · p. 4

IoU-balanced sampling

  • The two-stage re-sampling scheme balances positive and negative proposals and addresses their nonuniform duration-scale distribution.IoU-balanced sampling alone balances the positive/negative ratio, while the second stage adjusts sampling by proposal scale.Source: Re-sampling · p. 4, IoU-balanced sampling · pp. 4–5

Scale-balanced re-sampling

  • Scale-balanced re-sampling uses scale-wise positive ratios to set sampling probabilities across predefined normalized scale regions, applying the process to negative proposals too.The scale probability is adjusted for infrequently sampled regions; λ is set to 0.15 empirically.Source: Scale-balanced re-sampling · p. 5

Training and Inference of BSN++

Training combines the complementary boundary generator and proposal relation block objectives, with regularization, and balances proposal sampling. At inference, BSN++ fuses boundary and confidence maps to form proposal scores before suppressing redundant proposals.

Training

  • The overall loss combines the complementary boundary generator and proposal relation block objectives with L2 regularization.The weighting parameters β and γ are 10 and 0.0001, respectively.Source: Training · p. 5
  • Training truncates video features into sliding windows with 75% temporal overlap and labels start and end regions around ground-truth actions.Locations within either region receive the corresponding positive boundary label.Source: Training · p. 5
  • The proposal relation block predicts confidence maps and is trained with regression and binary-classification losses, using smooth-L1 for regression.Proposals with confidence values above 0.7 or below 0.3 are positive or negative, and two-stage sampling balances their scale and number near 1:1.Source: Training · p. 5

Inference

  • At inference, BSN++ fuses the boundary map with confidence maps by dot multiplication to form proposal scores, then applies Soft-NMS to suppress redundant proposals.The resulting candidates carry temporal start and end locations and scores.Source: Inference · p. 5

Experiments

BSN++ improves proposal-generation results on ActivityNet-1.3 and THUMOS14, and its proposals also boost detection results with the same classifiers. Ablations examine the model components, while an unseen-category evaluation reports only a slight performance drop.

Datasets and Setup

Temporal Proposal Generation

Ablation Experiments

  • Ablations associate the encoder-decoder with richer local and global context, bidirectional matching with future-context assistance, and the proposal relation block with more discriminative scoring.Scale-balanced sampling provides equivalent balancing; the integrated end-to-end model improves performance and efficiency.Source: Ablation Experiments · p. 6, Table 3 · p. 7
  • BSN++ shows only a slight performance drop on unseen ActivityNet-1.3 action categories, suggesting generalizability to unseen actions.The evaluation contrasts the Sports, Exercise, and Recreation and Socializing, Relaxing, and Leisure subsets.Source: Ablation Experiments · p. 7, Table 4 · p. 7

Action Detection with Our Proposals

Conclusion

BSN++ combines boundary prediction, proposal-relation modeling, and duration-aware balancing in a unified network. Experiments on ActivityNet-1.3 and THUMOS14 demonstrate effectiveness for temporal proposal generation and detection.

Conclusion

  • BSN++ combines a U-shaped complementary boundary generator with bi-directional boundary matching and a proposal relation block using two self-attention modules.The network generates boundary and confidence maps simultaneously; the paper also considers imbalanced proposal durations.Source: Conclusion · p. 7
  • Figure 5 shows qualitative BSN++ proposal examples from THUMOS14 and ActivityNet-1.3.Source: Figure 5 · p. 8
  • Experiments on ActivityNet-1.3 and THUMOS14 demonstrate BSN++'s effectiveness in temporal action proposal generation and detection.Source: Conclusion · pp. 7–8
Loading 2009.07641v5…