PaperlayerView exact revision on arXiv

Source-linked AI summary

G-TAD: Sub-Graph Localization for Temporal Action Detection

Mengmeng Xu, Chen Zhao, David S. Rojas, Ali Thabet, Bernard Ghanem

arXiv:1911.11462v2cs.CV

TL;DR

Temporal action detection needs context beyond temporal proximity, but prior methods primarily use temporal context. G-TAD formulates videos as graphs, dynamically aggregates multi-level semantic context with GCNeXt, and localizes action sub-graphs with SGAlign, achieving state-of-the-art performance on two benchmarks.

  • Problem

    Prior temporal action detection methods mainly use temporal context while neglecting semantic context and other desirable context properties.

  • Method

    G-TAD represents snippets as graph nodes, correlations as edges, and actions with context as sub-graphs, using GCNeXt and SGAlign for context aggregation and localization.

  • Results

    G-TAD achieves state-of-the-art performance on THUMOS-14 and ActivityNet-1.3, including 34.09% average mAP on ActivityNet-1.3.

  • Takeaways & Limitations

    G-TAD can find global, multi-level semantic video context adaptively without extra supervision.

Abstract

from arXiv · show

Temporal action detection is a fundamental yet challenging task in video understanding. Video context is a critical cue to effectively detect actions, but current works mainly focus on temporal context, while neglecting semantic context as well as other important context properties. In this work, we propose a graph convolutional network (GCN) model to adaptively incorporate multi-level semantic context into video features and cast temporal action detection as a sub-graph localization problem. Specifically, we formulate video snippets as graph nodes, snippet-snippet correlations as edges, and actions associated with context as target sub-graphs. With graph convolution as the basic operation, we design a GCN block called GCNeXt, which learns the features of each node by aggregating its context and dynamically updates the edges in the graph. To localize each sub-graph, we also design an SGAlign layer to embed each sub-graph into the Euclidean space. Extensive experiments show that G-TAD is capable of finding effective video context without extra supervision and achieves state-of-the-art performance on two detection benchmarks. On ActivityNet-1.3, it obtains an average mAP of 34.09%; on THUMOS14, it reaches 51.6% at IoU@0.5 when combined with a proposal processing method. G-TAD code is publicly available at https://github.com/frostinassiky/gtad.

1. Introduction

Temporal action detection must identify action boundaries and classes in long untrimmed videos, where useful context extends beyond temporal proximity. G-TAD models videos as graphs to adaptively aggregate multi-level semantic context and localize action sub-graphs.

  • Temporal action detection identifies action start and end frames and predicts class labels, especially challenging in long untrimmed videos.
  • Existing context-based methods mainly use temporal context, such as expanded boundaries, dilated convolutions, or Gaussian aggregation.
  • Desirable context should be semantically correlated, content-adaptive, and drawn from multiple semantic levels rather than merely nearby in time.
  • G-TAD represents snippets as graph nodes, snippet correlations as edges, and actions with context as target sub-graphs.
  • Temporal edges connect adjacent snippets, while semantic edges are dynamically learned from features so multi-level context is progressively aggregated.
  • 34.09% average mAP is achieved on ActivityNet-1.3, while 51.6% at IoU@0.5 is reached on THUMOS14 with proposal processing.

2. Related Work

Prior video-understanding work uses CNNs and graphs for action recognition and localization, but G-TAD differs by graphing snippets and combining temporal ordering with semantic similarity. Its architecture dynamically aggregates context before aligning and ranking candidate sub-graphs.

  • Action-recognition methods use two-stream 2D CNNs for RGB and optical flow or 3D CNNs for spatio-temporal encoding.
  • Graph-based video methods represent videos using spatial-temporal points, object-region proposals, or temporal action proposals, depending on the task.
  • G-TAD represents snippets as graph nodes and forms edges from both temporal ordering and semantic similarity, unlike prior graph formulations described here.
  • The architecture uses GCNeXt blocks to aggregate temporal and multi-level semantic context, SGAlign to create fixed-size sub-graph features, and localization to rank them.
  • Dynamic graph-convolution research includes feature-distance adjacency, deep residual or dense GCN connections, and dilated graph convolutions; G-TAD uses a DeepGCN-like structure.

3. Proposed Method

G-TAD represents video snippets as graph nodes and localizes action-associated sub-graphs using context-aware graph features and alignment. GCNeXt aggregates temporal and adaptive semantic context, while SGAlign produces fixed-size sub-graph representations for IoU prediction.

  • Problem Formulation: G-TAD averages consecutive frames into snippets and represents the resulting feature sequence as graph nodes for temporal action detection.Each annotated action specifies start time, end time, and class; predictions include temporal boundaries, class, and confidence.
  • GCNeXt for Context Feature Encoding: GCNeXt processes snippet features through temporal and semantic graph-convolution streams, then aggregates both streams with the input.The temporal stream uses fixed neighbors, while the semantic stream adaptively aggregates semantic context; each stream uses 32 split-transform-merge paths.
  • Sub-Graph Alignment and Localization: G-TAD applies fully connected layers to each SGAlign representation to predict its intersection-over-union with ground-truth actions.Training uses the maximum IoU over ground-truth actions as the sub-graph target, alongside node start/end classification losses.
  • GCNeXt for Context Feature Encoding: Temporal edges connect neighboring snippets, whereas semantic edges connect each node to dynamically selected nearest neighbors in feature space.Semantic neighbors are recomputed at every GCNeXt block, allowing semantic context to change across network levels.
  • Sub-Graph Alignment and Localization: SGAlign extracts fixed-size features from anchor-defined sub-graphs by interpolation and rescaling, using both GCNeXt and semantic features before concatenation.Semantic features aggregate neighbor information, so sub-graph representations incorporate multiple semantically correlated nodes.

4. Experiment

Experiments evaluate G-TAD on ActivityNet-1.3 and THUMOS14, compare it with existing detectors, ablate its components, and visualize how semantic context evolves.

  • Comparison with State-of-the-Art: G-TAD achieves the highest average mAP on ActivityNet-1.3 among the compared methods.
  • Comparison with State-of-the-Art: 51.6% mAP at IoU 0.5 is achieved on THUMOS14 when G-TAD is combined with P-GCN.
  • Comparison with State-of-the-Art: 23.4% mAP at IoU 0.7 exceeds TALNet’s 20.8%, while G-TAD alone trails TALNet at IoU 0.5.
  • Ablation Study: Semantic graph convolutions improve performance in both GCNeXt and SGAlign, with semantic context providing the main SGAlign gain.
  • Ablation Study: Larger THUMOS14 windows improve performance at L = 256, whereas L = 512 degrades performance because GPU memory reduces batch size.
  • Visualization: Semantic graphs develop more context edges across layers and training epochs, while first-layer edges remain fixed.

5. Conclusion

The conclusion presents G-TAD as a graph-based sub-graph localization approach that adaptively learns semantic context for temporal action detection and performs strongly on two benchmarks.

  • G-TAD formulates videos as graphs and temporal action detection as sub-graph localization.
  • GCNeXt aggregates context from semantically correlated snippets, while SGAlign converts sub-graph features into vector representations.
  • G-TAD learns multi-level semantic context adaptively with stacked dynamic graph convolutions and requires no extra supervision.
  • The experiments report state-of-the-art performance on THUMOS-14 and ActivityNet-1.3.

A. Derivation and Efficient Implementation of Eq. 4

The appendix derives Eq. 4 and explains an efficient implementation using zero-padded one-dimensional and edge convolutions.

  • The appendix provides the derivation of Eq. 4.
  • Eq. 4 is shown to admit an efficient implementation with zero-padded 1D convolutions.
  • The efficient implementation also uses zero-padded edge convolutions.

A.1. Derivation of Eq. 4

The appendix derives temporal graph convolution and notes that the same formulation extends to semantic graph convolution.

  • Temporal Graph Convolution: Temporal graph convolution is derived using forward and backward temporal edges between neighboring snippets.
  • Temporal Graph Convolution: The corresponding adjacency matrices represent temporal connections with vectors whose entries encode edge incidence.
  • Temporal Graph Convolution: Applying temporal graph convolution to X produces the output feature representation X_t.
  • Semantic Graph Convolution: The convolution uses trainable neural-network weights, and the same derivation applies straightforwardly to semantic graph convolution.

A.2. Efficient Implementation of Eq. 4

The implementation realizes temporal graph convolution with zero-padded 1D convolution and semantic graph convolution with edge convolution, while establishing equivalence to the formal operation.

  • Implementation: Zero-padded 1D convolution implements temporal graph convolution efficiently.The implementation uses zero padding at sequence boundaries.
  • Temporal Graph Convolution: For kernel size 3, each output combines the preceding, current, and following snippet features through three weight matrices.The output is defined for k = 1, 2, ..., L, with zero padding handling endpoints.
  • Equivalence: The convolution weights map directly to forward, backward, and self contributions in the graph formulation.The correspondence is W1 = Wb,1, W3 = Wf,1, and W2 = Wf,0 + Wb,0.
  • Equivalence: The derivation shows that the efficient convolution output equals the formal temporal graph-convolution expression.The proof concludes Xt = Y after substituting the corresponding trainable weights.
  • Semantic Graph Convolution: Semantic graph convolution uses edge convolution directly, so no analogous equivalence derivation is required.The passage states that the proof is therefore complete for the semantic graph case.

B. Training Details

The model combines semantic edges from multiple GCNeXt levels with SGAlign, while testing an IoU-map formulation using 2D convolutions and component ablations.

  • Semantic Edges from Multiple Levels: Multiple GCNeXt blocks adaptively incorporate multi-level semantic context into video features before SGAlign aligns sub-graph representations.SGAlign concatenates aligned features from temporal and semantic graphs.
  • SGAlign Components: SGAlign ablations show that semantic-graph context provides the main detection gain, while sample rescaling contributes only slight improvement.The experiment disables sample rescaling and semantic-graph feature concatenation on ActivityNet-1.3.
  • 2D Conv. for Sub-Graph Localization: The IoU-map variant arranges anchors on an L × L start/end-time grid and uses 2D CNNs to regress the corresponding map.Entries without pre-designed anchors are set to zero.
  • 2D Conv. for Sub-Graph Localization: Despite similar boundary locations among neighboring anchors, 2D convolutions provide no significant performance benefit.Kernel sizes 1, 3, and 5 were evaluated on ActivityNet-1.3.
Loading 1911.11462v2…