SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretization

Zhentao Tan; Ben Xue; Jian Jia; Junhao Wang; Wencai Ye; Shaoyun Shi; Mingjie Sun; Wenjin Wu; Quan Chen; Peng Jiang

Source-linked AI summary

SweetTok: Semantic-Aware Spatial-Temporal Tokenizer for Compact Video Discretization

Zhentao Tan, Ben Xue, Jian Jia, Junhao Wang, Wencai Ye, Shaoyun Shi, Mingjie Sun, Wenjin Wu, Quan Chen, Peng Jiang

arXiv:2412.10443v3Dec 11, 2024cs.CVcs.AI

TL;DR

Video tokenizers must reduce token counts without sacrificing reconstruction fidelity, but flattened or jointly compressed representations struggle with redundant spatial-temporal information and motion semantics. SweetTok separates spatial and temporal compression with DQAE and adds an MLC tailored to appearance and motion. It reports state-of-the-art gains across reconstruction and generation, while its semantic tokens support few-shot recognition but remain limited for precise semantic tasks such as VQA.

Problem
Video tokenizers need compact representations that preserve reconstruction quality despite redundant spatial-temporal information and insufficient video-specific language semantics.
Method
SweetTok uses DQAE for separate spatial-temporal query compression and MLC codebooks that assign different language parts of speech to appearance and motion information.
Results
SweetTok achieves state-of-the-art results across video reconstruction, image reconstruction, and class-conditional video generation, with reported margins of 42.8%, 37.2%, and 15.1%.
Takeaways & Limitations
The compressed decoupled tokens support downstream video generation and understanding, including few-shot action recognition with LLMs.
Takeaways & Limitations
SweetTok is not suitable for tasks requiring precise semantic understanding, such as VQA, because its MLC is trained unsupervised and image-text alignment remains challenging.

Abstract

from arXiv · show

This paper presents the \textbf{S}emantic-a\textbf{W}ar\textbf{E} spatial-t\textbf{E}mporal \textbf{T}okenizer (SweetTok), a novel video tokenizer to overcome the limitations in current video tokenization methods for compacted yet effective discretization. Unlike previous approaches that process flattened local visual patches via direct discretization or adaptive query tokenization, SweetTok proposes a decoupling framework, compressing visual inputs through distinct spatial and temporal queries via \textbf{D}ecoupled \textbf{Q}uery \textbf{A}uto\textbf{E}ncoder (DQAE). This design allows SweetTok to efficiently compress video token count while achieving superior fidelity by capturing essential information across spatial and temporal dimensions. Furthermore, we design a \textbf{M}otion-enhanced \textbf{L}anguage \textbf{C}odebook (MLC) tailored for spatial and temporal compression to address the differences in semantic representation between appearance and motion information. SweetTok significantly improves video reconstruction results by \textbf{42.8\%} w.r.t rFVD on UCF-101 dataset. With a better token compression strategy, it also boosts downstream video generation results by \textbf{15.1\%} w.r.t gFVD. Additionally, the compressed decoupled tokens are imbued with semantic information, enabling few-shot recognition capabilities powered by LLMs in downstream applications.

1. Introduction

SweetTok addresses the tension between compact video tokenization and reconstruction fidelity by decoupling spatial and temporal compression and adding motion-aware language semantics. It reports stronger reconstruction and generation results while using fewer tokens.

Motivation: Existing video tokenizers face a compression–fidelity trade-off because patch-based tokens retain redundant spatial and temporal location information.Flattening video tokens into a sequence can also make intertwined spatial-temporal information difficult to learn.
Method: SweetTok uses DQAE to compress spatial and temporal information into separate learnable queries before separate spatial and temporal decoding.The design targets heterogeneous redundancy in static images and dynamic frames and supports image reconstruction fine-tuning.
Method: MLC uses nouns and adjectives for static spatial information and verbs and adverbs for temporal motion information.The language-based codebooks are intended to address differing semantic representations across spatial and temporal information.
Results: SweetTok improves rFVD from 35.15 to 20.46 and gFVD from 99 to 84 versus LARP on UCF-101.Against OmniTok, it improves rFVD by 52.3% using only 25% of the tokens; on ImageNet-1k, rFID decreases from 0.59 to 0.37.
Results: 42.8% and 15.1% margins are reported for video reconstruction and class-conditional video generation, respectively, alongside a 37.2% image-reconstruction margin.The paper describes these as state-of-the-art results across video reconstruction, image reconstruction, and class-conditional video generation.

2. Background

Visual tokenizers discretize visual data for sequence prediction, while video tokenizers extend this idea with spatial-temporal modeling and compression. SweetTok builds on these approaches by designing a video-specific language codebook split across spatial and temporal attributes.

Visual Tokenization: Visual tokenizers convert visual inputs into discrete tokens that support generation and understanding as token-prediction tasks.The cited background describes visual tokenization as capturing essential temporal and spatial features for analysis.
Video Tokenizers: Video tokenizers use architectures including 3D-CNNs, ViTs with causal temporal transformers, and transformer-based image-pretrained models.TATS, CViViT, OmniTokenizer, and MAGVIT represent distinct approaches summarized in the background.
Query-Based Compression: LARP compresses flattened video patches into adaptive holistic queries guided by a pretrained autoregressive model.SweetTok inherits the broader spatial-temporal decomposition design for video data.
Language Codebooks: Prior language-codebook studies focus mainly on images, whereas SweetTok splits its video language codebook according to spatial-temporal attributes.This design targets the semantic differences between spatial and temporal video information.

3. Method

SweetTok addresses the difficulty of compacting video while preserving fidelity by separately compressing spatial and temporal information and enriching both with motion-aware language semantics.

3.2. Decoupled Query AutoEncoder (DQAE): SweetTok separates spatial and temporal quantization to avoid the intertwined redundancy and low-level content that hinder flattened-patch compression.The DQAE uses distinct learnable queries and combines the reconstructed components for video reconstruction.
3.2.2. Decoupled Query AutoEncoder (DQAE): Spatial tokenization encodes the first frame with learnable spatial queries, quantizes the resulting embeddings, and decodes reconstructed first-frame patches.The first frame is used because it usually contains most spatial information; the implementation sets L_spatial = 256.
3.2.2. Decoupled Query AutoEncoder (DQAE): Temporal tokenization compresses frame-wise residuals with temporal queries, then combines the quantized residuals with tiled reconstructed spatial patches to recover the video.This targets redundancy along the temporal dimension, with L_temporal = 1024 in the practical implementation.
3.2.2. Decoupled Query AutoEncoder (DQAE): The DQAE reconstructs latent video patches before a pixel decoder converts them back into video data, under reconstruction, perceptual, quantizer, and GAN losses.The pixel decoder is required because the reconstructed vectors remain in latent space.
3.2.3. Motion-enhanced Language Codebook (MLC): The Motion-enhanced Language Codebook uses nouns and adjectives for spatial appearance and verbs and adverbs for temporal motion.The codebook is built from caption vocabularies using CLIP embeddings and a graph convolution network that projects them into visual latent space.

4. Experiments

SweetTok is evaluated for compressed video reconstruction, generation, image reconstruction, ablations, and few-shot visual recognition. Across these settings, it combines strong fidelity with fewer tokens and shows benefits from decoupled queries and motion-enhanced language codebooks.

Video Reconstruction & Generation: 52.3% improvement in rFVD on UCF-101 is achieved with 1,280 tokens, four times fewer than OmniTok, while performance remains competitive on K-600.SweetTok uses 256 spatial and 1,024 temporal tokens; the comparison uses 5,120 OmniTok tokens.
Video Reconstruction & Generation: 68.8% and 28.5% rFVD improvements over similarly sized LARP-B are achieved on UCF-101 and K-600, respectively.Without token compression, SweetTok* reaches rFVD values of 10.74 on UCF-101 and 7.51 on K-600 at 5,120 tokens.
Video Reconstruction & Generation: 84 gFVD is achieved for class-conditional UCF-101 generation, 56% lower than OmniTok’s 191 and 15.1% better than LARP’s 99 at comparable generator sizes.Scaling the generator to 1.9B parameters further improves gFVD from 84 to 65.
Image Reconstruction: SweetTok improves ImageNet image reconstruction from TiTok’s rFID of 1.01 to 0.73 with 256 spatial tokens, while SweetTok* reaches 0.37 with 1,024 tokens.The 0.37 result surpasses LlammaGen-8 by 37.3%; visualizations indicate better global appearance and local details.
Ablations: Decoupled query compression outperforms naive interpolation and non-decoupled query compression, while the motion-enhanced temporal language codebook lowers rFVD from 24.80 to 20.46.Naive interpolation reaches rFVD 227.65, and training without decoupling reaches 35.15; vanilla language codebooks improve rFVD from 29.45 to 24.80.
Few-Shot Visual Classification: SweetTok tokens are converted into language words and used with CLIP and Qwen for few-shot image classification and video action recognition.The procedure selects the top 21 tokens by CLIP similarity to form the Qwen prompt.

5. Conclusions

SweetTok compresses spatial and temporal video information through a decoupled query autoencoder and combines this representation with a motion-enhanced language codebook. The resulting compact representation achieves higher reconstruction fidelity and supports downstream video generation and understanding.

5. Conclusions: SweetTok compresses spatial and temporal information through a decoupled query autoencoder, reducing video token count while improving reconstruction fidelity.Its compact representation is designed for downstream video generation and understanding.
5. Conclusions: The motion-enhanced language codebook captures action information in video and supports downstream video understanding tasks.The codebook complements the decoupled compression framework.

Discretization

The supplied passage lists notation explanations for the paper’s discretization framework.

Discretization: Table 7 provides explanations for the notations used in the main paper.

6. Experimental Settings

SweetTok is evaluated with a video tokenizer configured for 17-frame, 256 × 256 clips, alongside additional image and video benchmarks. The experiments use UCF-101, Kinetics-600, and ImageNet-1K, with training details reported separately.

Tokenizer configuration: SweetTok processes 17 consecutive 256 × 256 video frames and uses encoder-decoder stacks built from DQAE modules.The input is normalized to [−0.5, 0.5] before patchification into spatial and temporal features.
Evaluation metrics: Additional evaluation uses PSNR, SSIM, and LPIPS, with results reported in Table 8 for video and image datasets.The supplied passage states that SweetTok outperforms all baselines on these metrics.
Training setup: Video generation experiments use VideoGPT under the same protocol as the default large language model for pre-training.The supplementary implementation follows the protocol of [41].
Datasets: Experiments evaluate video tokenization on UCF-101 and Kinetics-600 and fine-tune the spatial component on ImageNet-1K for image tokenization.UCF-101 contains 13,320 videos across 101 actions, Kinetics-600 contains approximately 480K videos across 600 actions, and ImageNet-1K contains 1.2 million labeled images across 1,000 categories.
Training setup: SweetTok training uses 1 million steps, with proxy code applied during the first 500K steps and removed for the remaining 500K.Detailed hyperparameters are reported in Table 9.

7. Additional Results

Additional visual and semantic evaluations show that SweetTok reduces reconstruction differences across video and image datasets while producing meaningful motion-related language tokens. These results extend the quantitative evaluations with qualitative evidence for reconstruction and few-shot recognition.

Video reconstruction visualizations: SweetTok exhibits fewer pixel-level reconstruction differences than OmniTokenizer on UCF-101 and K-600.The visualizations use brighter regions for greater disparity and darker regions for consistency.
Image reconstruction visualizations: SweetTok outperforms all baselines by a significant margin in ImageNet-1K reconstruction visualizations.Differences are marked with red blocks, while green blocks show detailed regions.
Semantic word visualization: Correct verb words from the Motion-enhanced Language Codebook consistently lead to accurate UCF-101 action predictions even when other words are irrelevant.The visualization uses adjectives, nouns, adverbs, and verbs as prompts for Qwen LLM; green and orange indicate meaningful words, while red indicates meaningless ones.

8. Limitations

The supplementary visualizations compare reconstruction quality across video and image datasets, while the paper identifies a semantic-understanding limitation for tasks such as VQA. The limitation arises because the language codebook is trained without supervision aligning image and text domains.

Limitation: SweetTok is not suitable for precise semantic-understanding tasks such as VQA because its Motion-enhanced Language Codebook is trained unsupervised.The paper identifies contrastive learning between Qwen-VLM image features and codebook text embeddings as a possible remedy.
Video reconstruction visualizations: UCF-101 and K-600 visualizations compare OmniTokenizer and SweetTok using pixel differences from ground truth.The figure captions define “Diff” as the pixel difference between the ground truth and each model.
Image reconstruction visualization: The ImageNet-1K visualization compares TiTok, OmniTokenizer, and SweetTok, highlighting differences with red and green blocks.Red blocks select differences, while green blocks highlight corresponding details.
Semantic visualization: Few-shot UCF-101 action-recognition visualizations examine semantic words produced for video inputs.The figure caption identifies the visualization as a few-shot video action-recognition evaluation.