PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Query Networks for Fine-grained Video Understanding

Chuhan Zhang, Ankush Gupta, Andrew Zisserman

arXiv:2104.09496v1cs.CV

TL;DR

Fine-grained action recognition in untrimmed videos must distinguish subtle, sometimes brief events whose locations and durations are unknown during training. TQN answers predefined queries by attending to video features, with a stochastic feature bank supporting dense temporal sampling. The method surpasses state of the art on FineGym and Diving48 using only RGB features.

  • Problem

    Fine-grained actions in untrimmed videos differ in subtle details, while event duration and location vary and are unknown during weakly supervised training.

    Source: §1 · p. 2

  • Method

    TQN uses predefined queries to produce query-dependent responses from video features, while a stochastic feature bank supports dense sampling within finite GPU memory.

    Source: §1 · pp. 1–2

  • Results

    The TQN with stochastic feature-bank training surpasses state of the art on FineGym and Diving48 using only RGB features.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TQN provides temporal localization and alignment of semantically consistent events, and its query-response mechanism supports sharing training videos across common sub-labels.

    Source: §6 · p. 9

    The ST-S3D baseline makes decisions from individual clips that may not contain the relevant information.

    Source: §4 · p. 6

Abstract

from arXiv · show

Our objective in this work is fine-grained classification of actions in untrimmed videos, where the actions may be temporally extended or may span only a few frames of the video. We cast this into a query-response mechanism, where each query addresses a particular question, and has its own response label set. We make the following four contributions: (I) We propose a new model - a Temporal Query Network - which enables the query-response functionality, and a structural understanding of fine-grained actions. It attends to relevant segments for each query with a temporal attention mechanism, and can be trained using only the labels for each query. (ii) We propose a new way - stochastic feature bank update - to train a network on videos of various lengths with the dense sampling required to respond to fine-grained queries. (iii) We compare the TQN to other architectures and text supervision methods, and analyze their pros and cons. Finally, (iv) we evaluate the method extensively on the FineGym and Diving48 benchmarks for fine-grained action classification and surpass the state-of-the-art using only RGB features.

1. Introduction

Fine-grained recognition in untrimmed videos must distinguish subtle actions whose locations and durations are unknown, sometimes from only a few frames. The paper introduces a query-based model and stochastic feature-bank training, and reports surpassing state of the art on FineGym and Diving48 using RGB features.

  • The Temporal Query Network maps a video and predefined queries to query-dependent responses, trained with labels for each query rather than temporal localization.Source: §1 · p. 1
  • Queries examine the video to find relevant segments and ignore irrelevant content; whole-video average pooling may dilute signals from events lasting only a few frames.Source: §1 · pp. 1–2
  • The stochastic feature bank caches densely sampled contiguous features and computes and backpropagates through a random subset each iteration, accommodating finite GPU memory.Source: §1 · p. 2
  • Fine-grained events may be short, subtle, and variably located within long videos, while training lacks their temporal alignment.The datasets include events around 0.3 seconds in clips typically 6–10 seconds long, sometimes reaching 30 seconds.Source: §1 · p. 2
  • The TQN with stochastic feature-bank training surpasses state of the art on FineGym and Diving48 using only RGB features.Source: §1 · p. 2

2. Related Work

Prior work spans action-recognition architectures, long-term video modeling, and video-text supervision. TQN uses query-response attention for action recognition, with dataset-wide predefined queries rather than video-question answering.

  • Action-recognition architectures have improved performance, but fine-grained recognition still faces background bias, long-term understanding, and subtle action distinctions.Source: §2 · p. 2
  • Long-term video work includes RNN context modeling and Transformer-based approaches, often combining Transformers with ConvNets for video tasks.Source: §2 · p. 2
  • Video-text methods use captions, sparse labels, or free-form sentences for localization, while TQN answers predefined questions with a shared response set.Source: §2 · p. 2
  • Like video-question-answering models, TQN uses query-response attention, but its queries are shared across the dataset and its goal is action recognition.Source: §2 · pp. 2–3, Figure 2 · p. 3

3. Method

The method pairs pre-defined queries with query-specific response sets, using a Transformer decoder to attend over densely sampled untrimmed videos under weak supervision. It also factorizes fine-grained categories into attribute queries and uses a stochastically updated feature bank to train over extended temporal context.

3.1. Temporal Query Networks

  • The TQN uses learned, semantically distinct queries to attend over clip features and produce query-specific responses without temporal localization supervision.Its parallel Transformer decoder refines query responses through query self-attention, visual-feature attention, and feed-forward layers.Source: §3.1 · pp. 3–4
  • The visual encoder, TQN decoder, and attribute classifiers are trained jointly using a multi-task combination of per-query softmax cross-entropy losses.Each loss uses the ground-truth attribute for its label query.Source: §3.1 · p. 4

3.2. Stochastically Updated Feature Bank

  • The feature bank caches clip features, replacing randomly sampled contiguous clips with online encoder outputs while retrieving the remaining features from the bank.The combined features feed the TQN for prediction and backpropagation; the bank is refreshed with the online features.Source: §3.2 · p. 4
  • The feature bank decouples video length from the GPU memory budget and supports joint encoder-decoder training with extended temporal context.It also allows multiple videos in a mini-batch rather than only one long video.Source: §3.2 · p. 4

3.3. Factorizing Categories into Attribute Queries

  • Category labels are factored into semantic sub-label queries and attribute sets, allowing shared video data across common sub-labels.The categories are represented as a subset of the Cartesian product of the attribute sets.Source: §3.3 · p. 5

3.4. Implementation Details

  • The implementation uses an S3D backbone and a two-stage curriculum, ending with memory-bank training on the whole training set.Stage one trains on short videos; stage two uses the stochastically updated memory bank to accommodate long videos.Source: §3.4 · p. 6

4. Datasets, Baselines, Label Sets, and Metrics

The evaluation covers fine-grained recognition on FineGym and Diving48-v2, with datasets, baselines, query-attribute labels, and accuracy measures specified. The training setup uses dense temporal support without temporal localization labels.

Datasets

  • FineGym evaluation uses Gym99 and Gym288, while Diving48-v2 covers 48 competitive-diving classes with subtle sequence differences and video lengths ranging from 24 to 822 frames.Gym99 is relatively balanced and Gym288 is long-tailed; Diving48-v2 uses cleaned-up labels.Source: §4 · p. 6

Baselines

  • The baselines are short-term S3D on fixed eight-frame clips and long-term S3D with a feature bank pooling information across multiple clips.For ST-S3D, multi-clip probabilities are averaged at inference; LT-S3D also uses the multi-clip setting during training.Source: §4 · p. 6
  • ST-S3D training bases decisions on individual clips that may lack classification-relevant information, whereas LT-S3D uses a multi-clip feature bank.The paper says LT-S3D learns better clip-level features through this setting.Source: §4 · pp. 6–7

Label Sets

  • Diving48 uses four stage queries, while FineGym descriptions are factored into 12 queries and attributes, with an additional global query for final category prediction.The attributes instantiate each dataset’s query set; the global query uses original fine-grained categories as its attributes.Source: §4 · p. 7

Metrics

  • Top-1 classification accuracy is measured per original class and per video.The original-class counts are 48 for Diving48, 99 for Gym99, and 288 for Gym288.Source: §4 · p. 7

5. Experiments

Across the reported benchmarks, TQN improves on state-of-the-art methods, while ablations show benefits from multi-attribute supervision and long, updated feature banks. Its attention has advantages on longer videos, and it also performs comparably to state-of-the-art methods on Charades.

5.1. Leveraging Multi-Attribute Labels

  • 81.8% per-video accuracy: TQN multi-task losses outperform standard multi-class alternatives and sequence-based text methods on Diving48-v2.Standard multi-class results are 80.4% with average pooling and 80.0% with self-attention; sequence-based text methods perform 15–30% worse.Source: §5.1 · p. 7

5.2. Feature Bank Ablations

  • 80.5% per-video accuracy with all frames versus <75% for N={8, 32, 64}; updating bank features improves performance by approximately 15% with or without the TQN decoder.Longer temporal context helps across sampling strategies, and the feature-bank comparison is on Diving48-v2.Source: §5.2 · p. 7, Table 5 · p. 8, Table 4 · p. 8

5.3. Comparison with State-of-the-art

  • TQN outperforms all compared methods on Diving48-v2, Gym99, and Gym288 for per-video and per-class measures, using RGB while some alternatives use RGB and flow.Long-term S3D improves over short-term S3D by more than 30% absolute on Diving48-v2 and more than 10% on the Gym datasets; adding TQN yields further gains.Source: §5.3 · p. 8, Table 3 · p. 8

5.4. Performance on Videos of Different Length

  • For videos longer than five seconds, TQN attention outperforms LT-S3D; Gym288-pretrained fine-tuning improves Diving48-v2 accuracy and trains faster than training from scratch.On videos shorter than five seconds, LT-S3D performs similarly to TQN.Source: §5.4 · p. 8, Figure 6 · p. 8, §5.5 · p. 9

5.6. Extension to Multi-label Action Recognition

  • On Charades, TQN achieves results comparable to state-of-the-art models for multi-label action recognition.Unlike FineGym and Diving48, Charades labels multiple actions separately and provides precise temporal annotations.Source: §5.6 · p. 9

6. Conclusion

The conclusion presents TQN as a model for fine-grained event and attribute questions in untrimmed videos. It reports state-of-the-art fine-grained categorization results on three datasets and identifies broader dense-stream applications.

Conclusion

  • TQN answers fine-grained questions about event types and attributes in untrimmed videos.The conclusion describes it as a video parsing model.Source: §6 · p. 9
  • TQN advances state-of-the-art fine-grained video categorization on three datasets and provides temporal localization and alignment of semantically consistent events.Source: §6 · p. 9
  • The authors identify spotting entities with varying spans in dense data streams as a broader application, with stochastic feature-bank updates avoiding heavy large-scale training infrastructure requirements.Source: §6 · p. 9

Appendices

The appendices visualize query attention, report RGB-versus-flow comparisons, and extend TQN to multi-label action recognition on Charades.

A. Visualization of Action Attention

B. Full SotA Comparison

  • On FineGym, TQN achieves superior results to prior methods using only RGB frames, including methods whose reported results use both RGB and flow.The appendix provides the RGB and flow breakdown for the state-of-the-art comparison.Source: B. Full SotA Comparison · p. 14, Table 7 · p. 14

C. Extension to Multi-label Action Recognition

  • On Charades, the stochastically updated feature bank further improves TQN’s multi-label performance, producing better mAP than other models with the same Kinetics-400-pretrained visual backbone.Each of 157 action classes is treated as an individual query with a binary output; training uses short clips followed by the dense feature bank for video-level prediction.Source: C. Extension to Multi-label Action… · p. 14

D. Implementation Details

The implementation uses a Transformer decoder that treats learned query embeddings as queries over visual features, then classifies the updated query representations. Training and inference settings vary by dataset, with a two-stage, stochastically updated feature bank supporting long videos.

D. Implementation Details

  • The decoder uses learned query embeddings to attend to visual features as keys and values, then applies linear classifiers to its output features.The decoder has the architecture of a basic Transformer decoder.Source: D. Implementation Details · p. 14
  • FineGym and Diving48 training uses spatial crops and removes 2% of frames at random, while testing averages predictions from three sequences with 2% frame removal.Testing uses center crops; Charades instead uses three spatial crops and max-pools their predictions.Source: D. Implementation Details · p. 15
  • FineGym and Diving48 use a two-stage pipeline: end-to-end warm-up on short videos, followed by training with a stochastically updated feature bank on longer videos.No temporal localization information is used in training for these datasets.Source: D. Implementation Details · p. 15
  • The appendix provides dataset-specific training schedules and reports a Charades comparison using temporal localization.Table 9 details multi-stage feature-bank training across datasets; Table 8 compares Charades results when temporal localization is used.Source: D. Implementation Details · p. 15, Table 8 · p. 15, Table 9 · p. 15
  • During feature-bank training, randomly selected contiguous clips are computed online and cached; all video features are computed online at inference.The online update uses 10 clips for Diving48, 6 for FineGym, and 3 for Charades.Source: D. Implementation Details · p. 15

E. Ablation on Two-Stage Training

On Diving48, both warm-up length and the number of online-updated clips affect performance. The authors conclude that both training stages contribute: warm-up reduces overfitting to uninformative clips, while feature-bank training handles long sequences.

E. Ablation on Two-Stage Training

  • Increasing kfirst to 8 improved both per-video and per-class accuracy by 2% over the shorter warm-up setting.With kfirst=0, the pretrained initialization yielded 72.3% per-class and 79.2% per-video accuracy; warm-up on videos shorter than four clips worsened performance because they represent only 5% of the dataset.Source: E. Ablation on Two-Stage Training · p. 16, Table 10 · p. 16
  • Doubling nonline from 4 to 8 improved both per-class and per-video accuracy by 2%.The experiment fixes kfirst at 8; nonline=0 denotes no second-stage memory-bank training.Source: E. Ablation on Two-Stage Training · p. 16
  • Both stages are needed for good results: warm-up helps prevent overfitting to uninformative clips, while the second stage supports training on sequences too long for the first stage.The authors note that larger kfirst and nonline could improve performance with more GPU memory.Source: E. Ablation on Two-Stage Training · p. 16

F. Feature Bank vs. End-to-end Training

The feature bank approaches end-to-end performance when initialized well and updated with large steps, while text-supervision comparisons show that the query-based factorization outperforms the described alternatives. The section also reports weaker results for single-head multi-label and autoregressive sequence prediction.

F. Feature Bank vs. End-to-end Training

  • The feature-bank result with 12 updated clips was less than 1% below end-to-end training on videos shorter than 256 frames.The comparison uses the feature bank with different update step sizes; the authors state that good initialization and a large update step can bring its performance close to end-to-end training.Source: F. Feature Bank vs. End-to-end Training · p. 16, Table 12 · p. 16
  • The main-paper Diving48 setting updates 10 clips, or 160 frames (50% of maximum length), and reaches 81.8% per-video accuracy.The authors note that more GPU memory and a larger update step might improve accuracy further.Source: F. Feature Bank vs. End-to-end Training · p. 16

G. Text Supervision

  • The S3D average-pooling baseline reaches 72.3% per-class and 80.4% per-video accuracy; adding a Transformer encoder increases per-class accuracy by 1.4%.The baseline applies average pooling and a linear classifier, while the encoder variant adds self-attention before temporal average pooling.Source: G. Text Supervision · p. 17
  • A single-head multi-label model has 25% lower per-video accuracy than the multi-class models, and autoregressive sequence prediction is 16% lower than TQN.The multi-label setup predicts attributes with one linear head; the sequence model's next-step dependence suits grammatical language but not the task's discretized, independent action types.Source: G. Text Supervision · pp. 17–18
  • The sequence-prediction alternative decodes the ground-truth attributes autoregressively and selects the class whose ground-truth sequence has the smallest edit distance from its output.The compared methods use S3D as the visual encoder, with Transformer components set to four attention layers and heads when present.Source: G. Text Supervision · p. 17

H. Comparison to SotA Results on Diving48-v1 (the noisy version)

The authors treat Diving48-v1 results cautiously because the early release was heavily mislabeled. On that version, TQN improves over the baseline but ranks second; the comparison does not establish why the deepest model performs best.

H. Comparison to SotA Results on Diving48-v1 (the noisy version)

  • Diving48-v1 contains mislabeled 43.5% of training and 35.8% of test videos, so the authors report main-paper comparisons on corrected Diving48-v2 instead.They say previous Diving48-v1 results are hard to interpret given the label noise.Source: H. Comparison to SotA Results on… · p. 18
  • On noisy Diving48-v1, the query decoder improves over LT-S3D by 2% and ranks second in the comparison.CorrNet-101 has the best result, but its advantage may reflect better fitting of label noise rather than better action understanding.Source: H. Comparison to SotA Results on… · p. 18, Table 14 · p. 18

I. Error Analysis: Confusion Matrices

The confusion matrices show strong prediction of pose-based attributes, while errors concentrate on counting turns and twists and on attributes with very few training examples. The query-attribute tables specify the response sets and class mappings for Diving48 and Gym99.

I. Error Analysis: Confusion Matrices

  • TQN does well at predicting pose-based attributes for ‘take off’, ‘flight pose’ and ‘position’.The confusion matrices cover Diving48 and Gym99.Source: §I · p. 19, Figure 9 · p. 19, Figure 10 · p. 20
  • The main errors involve counting turns and twists, especially when their counts are similar.The ‘3.5Twis’ attribute for the ‘twist’ query accounts for only 0.3% of training data.Source: §I · p. 19

J. Query-Attributes Specification

Related papers

Loading 2104.09496v1…