PaperlayerView exact revision on arXiv

Source-linked AI summary

TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding

Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, Lu Hou

arXiv:2310.19060v1cs.CVcs.AIcs.CL

TL;DR

Long-form video encoding is costly because videos contain massive, redundant visual-token sequences, and existing approaches can limit temporal coverage or full fusion. TESTA progressively aggregates similar frames and patches in a divided space-time scheme. The model reports 1.7 times greater computing efficiency and performance gains on longer inputs.

  • Problem

    Long-form video-language understanding faces a costly visual-token burden, while sparse sampling and offline encoding can limit temporal information or end-to-end feature fusion.

    Source: §1 · p. 1

  • Method

    TESTA progressively aggregates similar frames and within-frame patches during encoding, with temporal and spatial aggregation modules in each video encoder block.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    TESTA improves computing efficiency by 1.7 times while maintaining comparable performance, and reports +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie with more input frames.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TESTA’s aggregation condenses video semantics while retaining frame and patch features that can correspond to objects, scenes, and events.

    Source: §1 · p. 2

    Because of limited computing resources, the authors do not use long-form video pre-training datasets or incorporate TESTA into pre-training.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.

1 Introduction

Long-form video-language understanding is constrained by the cost of processing redundant visual tokens, while sparse sampling and offline encoding limit temporal coverage or end-to-end fusion. TESTA aggregates similar frames and patches during encoding, reducing token count while retaining temporal-spatial modeling.

1 Introduction

  • Sparse sampling reduces visual-token counts but sacrifices temporal dynamics and storyline information; offline encoding can limit task adaptation and full fusion of frame features.The paper frames sufficient input frames and end-to-end temporal-spatial modeling as important for optimal performance.Source: §1 · p. 1
  • TESTA samples frames densely, then separately aggregates similar frames and within-frame patches during video encoding.It averages matched features and can reduce visual tokens by around 75%.Source: §1 · p. 2, Figure 1 · p. 2
  • The method is inspired by Token Merging (ToMe) (Bolya et al., 2022), but its divided temporal and spatial aggregation reduces computation complexity.The aggregated tokens can correspond to higher-level semantics such as objects, scenes, and events, and may be grounded in language.Source: §1 · p. 2
  • 1.7× greater computing efficiency with comparable performance, plus +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie with more input frames, are reported for TESTA.The latter gains are compared with previous state-of-the-art methods.Source: §1 · p. 2

2 Related Work

Related work covers video-language pre-training and efficient video Transformers, then positions TESTA as a unified approach that progressively aggregates both frame and patch features during encoding.

2 Related Work

  • Video-language pre-trained models encode video-text pairs into a shared feature space for semantic alignment and may add a text decoder for captioning or VideoQA (Zhang et al., 2020).The encoder-decoder pattern is described alongside prior work (Chen et al., 2022; Sun et al., 2022; Cheng et al., 2022).Source: §2 · pp. 2–3
  • Efficient Transformer designs factorize spatial and temporal attention (Arnab et al., 2021) or restrict joint attention to local 3D windows (Liu et al., 2021).TESTA is presented as orthogonal: it aggregates spatial and temporal token features to reduce each block’s input size.Source: §2 · p. 3
  • Earlier aggregation work models time through post-encoding frame aggregation and space through patch merging or proxy tokens, whereas TESTA unifies frame and patch aggregation during encoding (Bain et al., 2022; Luo et al., 2021; Bolya et al., 2022; Shi et al., 2023; Cao et al., 2023; Xu et al., 2022; Ryoo et al., 2021; Marin et al., 2021).The paper says progressive aggregation improves efficiency while retaining feature interaction in both space and time.Source: §2 · p. 3

3 Method

The method combines a divided space-time video encoder with progressive temporal and spatial token aggregation. TESTA defines frame and patch tokens, matches and merges similar tokens, and uses three video-text pre-training objectives.

3.1 Model Architecture

  • Each video encoder block applies temporal attention across frames and spatial attention within each frame, with aggregation modules added to improve encoding efficiency.The broader model uses three encoders and one decoder; its text and video-grounded text components support unimodal encoding, cross-modal information flow, and autoregressive generation.Source: §3.1 · pp. 3–4

3.2 Temporal-Spatial Token Aggregation

  • TAM reduces the frame count and SAM reduces the patch count progressively across encoder blocks, shrinking the features processed by subsequent blocks.The reduction amounts RT and RS are hyperparameters controlling the performance-efficiency tradeoff.Source: §3.2 · p. 4

3.2.1 Objects for Aggregation

  • TESTA treats image patches as patch tokens and represents each frame with a pseudo frame token obtained by averaging its patch tokens.The same aggregation strategy applies to both token types.Source: §3.2.1 · p. 4

3.2.2 Aggregation Strategy

  • Bipartite matching partitions tokens into sets, pairs each reduction-set token with its most similar counterpart, and merges matched features by averaging.Similarity uses cosine similarity between attention keys; the algorithm is parameter-free and used during training and testing.Source: §3.2.2 · pp. 4–5
  • Importance-based aggregation selects the least-important tokens using received attention, while geometry-based partitioning separates adjacent tokens so they can be matched across sets.The attention score is defined from attention received from other tokens; geometry-based aggregation addresses adjacent tokens being grouped together by the importance-based strategy.Source: §3.2.2 · p. 5

3.2.3 Novelty over Token Merging

  • Unlike ToMe’s joint space-time tokens and global aggregation, TESTA uses separate frame and patch tokens and divided temporal-spatial aggregation.The paper states that this reduces complexity and allows efficient encoding of more than 128 frames under the same computation quota.Source: §3.2.3 · p. 5

3.3 Pre-training Objectives

  • Pre-training uses video-text contrastive, video-text matching, and captioning objectives.These are identified as the paper’s three classic pre-training objectives.Source: §3.3 · p. 5

4 Experiments

TESTA is evaluated on paragraph-to-video retrieval and long-form VideoQA, with comparisons covering different frame counts, pre-training settings, and aggregation choices. Results show strong retrieval and VideoQA performance, while ablations examine efficiency, aggregation strategy, and how temporal and spatial aggregation work together.

4.1 Implementation Details

  • For fine-tuning, TESTA uses 32 or 96 frames at 224 × 224 resolution, with aggregation settings chosen for approximately 50% lower computation cost.The model starts from a BLIP checkpoint and is pre-trained on around 5M image-text and video-text pairs.Source: §4.1 · p. 5

4.3 Paragraph-to-Video Retrieval

  • At 96 frames, TESTA reaches 83.4 R@1 on QuerYD, 24.9 R@1 on Condensed Movie, 61.2 R@1 on DiDeMo, and 54.8 R@1 on ActivityNet.On DiDeMo and ActivityNet, these results surpass previous SOTA methods by 6.6 and 1.4, respectively.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · p. 7

4.4 Long-Form Video Question-Answering

4.2 Downstream Task Setups

  • Evaluation covers paragraph-to-video retrieval on four datasets and long-form VideoQA on ActivityNet-QA.Source: §4.2 · p. 6

4.5 Zero-shot Generalizability

  • In zero-shot evaluation with 32 frames, TESTA outperforms BLIP by average improvements of +14.1 on QuerYD, +2.9 on DiDeMo, and +3.8 on ActivityNet.The authors attribute the gains to temporal modeling and pre-training on video data, not solely to the BLIP checkpoint initialization.Source: Table 4 · p. 7, §4.5 · p. 7

4.6 Ablation Study

  • On QuerYD with 96 frames, aggregation degrades average recall less than pruning; geometry-based aggregation leads the tested strategies, and combining temporal and spatial aggregation performs best.Average-recall degradation is −2.2 for aggregation versus −8.4 for pruning; geometry-based aggregation reaches 83.4 Recall@1, 3.2 above the other strategy.Source: Table 5 · p. 7, §4.6 · p. 8

4.7 Comparison to Token Merging

  • On QuerYD with 16 frames and matched GFLOPs, TESTA achieves higher recall with fewer GFLOPs than ToMe, with or without token aggregation.Source: Table 6 · p. 9, §4.7 · p. 9

4.8 Visualization

  • Visualizations show temporal merging of similar frames, spatial merging of patches from shared scenes or objects, and language grounding to aggregated regions.Phrase queries achieve their highest similarity in the corresponding oracle regions formed by aggregation.Source: Figure 4 · p. 8, §4.8 · p. 9

5 Conclusion

TESTA condenses video semantics by aggregating similar frames and patches, accelerating video encoding. On paragraph-to-video retrieval and VideoQA, it outperforms previous state-of-the-art methods by a considerable margin.

5 Conclusion

  • TESTA outperforms previous state-of-the-art methods by a considerable margin on paragraph-to-video retrieval and VideoQA tasks.The method aggregates similar frames and patches to condense video semantics and accelerate encoding.Source: §5 · p. 9

Limitations

The paper identifies limitations in pre-training data and token aggregation, as well as the coarseness of its training objectives. Its pre-training datasets comprise WebVid-2M and CC3M, with no token aggregation during pre-training.

Limitations

  • The model is not pre-trained on long-form video datasets or with TESTA, which the authors suggest could improve pre-training efficiency and model performance.The authors also identify using text signals for aggregation as a possible way to make encoded features more suitable for downstream tasks.Source: Limitations · p. 9
  • Training uses coarse objectives on video-text pairs; the authors propose fine-grained alignment as a direction for tasks such as action localization and video object detection.They note that TESTA can aggregate tokens into objects, scenes, and events.Source: Limitations · p. 9

A.1 Pre-training Datasets.

  • Pre-training uses WebVid-2M video-text pairs and CC3M image-text pairs; token aggregation is omitted because the pre-training videos have relatively few frames.The datasets contain 2.5M and 3M pairs, respectively; unavailable URLs bring the total number of pre-training samples to around 5M.Source: §A.1 · p. 12

A.2 Detailed Pre-training Objectives.

The pre-training objectives combine video-text contrastive learning, video-text matching, and caption generation. The appendices specify pre-training and fine-tuning settings, including the retrieval and VideoQA evaluation procedures.

A.2 Detailed Pre-training Objectives.

  • Video-text contrastive learning pulls paired video and text features together while pushing apart semantically dissimilar pairs.The implementation uses negative features from a recent-sample queue and adds momentum distillation regularization.Source: §A.2 · pp. 12–13
  • Video-text matching predicts whether a video-text pair matches using a joint [ENC] embedding and a two-class probability.The implementation applies online contrastive hard negative mining.Source: §A.2 · p. 13
  • Captioning loss trains the video-grounded text decoder to predict the tokenized caption autoregressively.The overall objective combines the three pre-training losses.Source: §A.2 · p. 13

A.3 Hyperparameters.

  • Pre-training runs for 5 epochs with Adam, batch size 384, and a momentum queue of 57,600 on four nodes with 32 NVIDIA V100 GPUs.Each epoch lasts around 6 hours; learning rate warms up from 1e-6 to 5e-6, then cosine decays to 5e-7.Source: §A.3 · p. 13

B Fine-tuning Details

  • Retrieval fine-tuning uses contrastive and matching objectives, then reranks the top 128 similarity-selected candidates by pairwise matching scores.For VideoQA, the model is trained with cross-entropy and candidates are ranked by generation probability.Source: B Fine-tuning Details · p. 13

C Downstream Datasets

The downstream evaluation covers paragraph-to-video retrieval and long-form VideoQA. For QuerYD, the figure compares GFLOPs with Recall@1 and shows TESTA in the upper-left performance-cost region.

C Downstream Datasets

  • Paragraph-to-video retrieval is evaluated on DiDeMo, QuerYD, ActivityNet Captions, and Condensed Movie using text-to-video R@k.R@k is the recall percentage of the target video within the top K retrieval results.Source: C Downstream Datasets · p. 14
  • Long-form VideoQA is evaluated on ActivityNet-QA using accuracy.Source: C Downstream Datasets · p. 14
  • TESTA occupies the upper-left region of the QuerYD GFLOPs-versus-Recall@1 figure, indicating a better performance-cost tradeoff than other pre-trained models.The figure distinguishes input-frame counts using nF, where n is the number of frames used for fine-tuning and evaluation.Source: Figure 5 · p. 14

D Recall-GFLOPs Tradeoff of Various Pre-trained Models

TESTA has a superior Recall-GFLOPs tradeoff among the compared pre-trained models. Unlike several baselines, its performance improves linearly as more input frames are used.

D Recall-GFLOPs Tradeoff of Various Pre-trained Models

E Ablation on the Number of Reduced Tokens

The ablation shows that reducing more tokens lowers computation, but excessive aggregation can reduce recall. The selected settings cut computation by about half without significant performance decline.

E Ablation on the Number of Reduced Tokens

  • GFLOPs decrease linearly as R6 increases, but merging too many tokens, such as RT = 10, loses semantic information and lowers average recall.The ablation varies RT and RS and reports average GFLOPs and recall on QuerYD.Source: E Ablation on the Number of Reduced… · p. 14
  • Approximately 50% computation reduction is achieved without significant performance decline using the selected TESTA configuration.The defaults are RT = 4 and RS = 8 for 96-frame inputs, and RT = 1 and RS = 12 for 32-frame inputs.Source: E Ablation on the Number of Reduced… · p. 14
  • For 96-frame inputs, the default is RT = 4 and RS = 8; for 32-frame inputs, it is RT = 1 and RS = 12.The defaults were selected based on GFLOPs-recall tradeoff results.Source: E Ablation on the Number of Reduced… · p. 14

F Token Similarity Analysis

Token-similarity analysis finds substantial patch-token similarity and high similarity among aggregated patch tokens, while frame-token representations remain a stated limitation. DiDeMo visualizations show TESTA aggregating semantically similar tokens.

F Token Similarity Analysis

  • Overall patch-token similarity exceeds 0.5, while aggregated patch tokens reach 0.96 similarity, supporting their semantic purity.The analysis compares token similarities across encoder blocks, token dimensions, and aggregation status.Source: F Token Similarity Analysis · p. 14
  • Frame-token similarity decreases across blocks and may yield aggregated frames with mixed semantics; the authors identify frame-token representation and similarity as needing improvement.Frame tokens are pseudo tokens obtained by averaging patch features, rather than an elaborate model of frame semantics.Source: F Token Similarity Analysis · p. 15

G More Visualization of Aggregation

  • DiDeMo visualizations show TESTA aggregating tokens with highly similar semantics; shared rectangles mark merged frames, and matching patch colors mark merged patches.The passage describes four additional qualitative cases and characterizes the visualizations as interpretable.Source: G More Visualization of Aggregation · p. 15, Figure 9 · p. 16

Related papers

Loading 2310.19060v1…