PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer

Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Ping Luo, Wanli Ouyang, Xiaogang Wang

arXiv:2204.08680v3cs.CV

TL;DR

Fixed-grid vision tokens treat image regions uniformly, although human-centric tasks require fine representation of important regions and fewer tokens for backgrounds. TCFormer progressively clusters and merges tokens into flexible, semantically adaptive regions, with multi-stage token aggregation preserving detail. It consistently outperforms counterparts across challenging human-centric tasks and datasets.

  • Problem

    Fixed-grid tokens cannot allocate flexible locations, shapes, sizes, or denser representations to important regions in human-centric vision.

  • Method

    TCFormer progressively clusters semantically similar tokens, merges each cluster, and aggregates token features across stages with its CTM and MTA components.

  • Results

    TCFormer consistently outperforms its counterparts on whole-body pose estimation and 3D human mesh reconstruction, among other human-centric tasks.

  • Takeaways & Limitations

    Flexible token regions let TCFormer focus computation on important areas while representing backgrounds more coarsely and preserving detailed information.

  • Takeaways & Limitations

    The quadratic complexity of KNN-DPC limits TCFormer's speed at large input resolutions.

Abstract

from arXiv · show

Vision transformers have achieved great successes in many computer vision tasks. Most methods generate vision tokens by splitting an image into a regular and fixed grid and treating each cell as a token. However, not all regions are equally important in human-centric vision tasks, e.g., the human body needs a fine representation with many tokens, while the image background can be modeled by a few tokens. To address this problem, we propose a novel Vision Transformer, called Token Clustering Transformer (TCFormer), which merges tokens by progressive clustering, where the tokens can be merged from different locations with flexible shapes and sizes. The tokens in TCFormer can not only focus on important areas but also adjust the token shapes to fit the semantic concept and adopt a fine resolution for regions containing critical details, which is beneficial to capturing detailed information. Extensive experiments show that TCFormer consistently outperforms its counterparts on different challenging human-centric tasks and datasets, including whole-body pose estimation on COCO-WholeBody and 3D human mesh reconstruction on 3DPW. Code is available at https://github.com/zengwang430521/TCFormer.git

1. Introduction

Human-centric vision requires finer, semantically adaptive representation of important regions than fixed-grid tokens provide. TCFormer progressively clusters tokens into flexible regions, preserves multi-stage detail, and improves performance across challenging human-centric tasks.

  • Motivation: Fixed-grid tokens allocate uniform rectangular regions, despite human bodies being more important than backgrounds in human-centric analysis.This limits allocation of additional tokens to important areas.
  • Token Clustering Transformer: TCFormer progressively clusters tokens to generate image-specific regions with varied locations, shapes, and sizes.Fine tokens represent detailed regions such as faces, while one token can cover large background areas.
  • Clustering-based Token Merge: CTM initializes pixels as tokens, groups them by semantic similarity, merges each cluster by averaging, and aggregates the merged tokens with a transformer block.The merged token region is the union of its input token regions.
  • Multi-stage Token Aggregation: MTA progressively upsamples and aggregates token features from all stages to preserve details that may be lost when dynamic tokens are mapped to feature grids.Its token-based aggregation avoids averaging multiple detailed tokens into the same low-resolution pixel grid.
  • Results: TCFormer consistently outperforms its counterparts on challenging human-centric tasks including COCO-WholeBody whole-body pose estimation and 3DPW 3D human mesh reconstruction.The paper also introduces MTA to aggregate multi-stage token features efficiently.

2. Related Works

Prior human-centric vision transformers use fixed-grid tokens or modify grid sampling, limiting token regions to handcrafted structures. TCFormer instead learns flexible regions through semantic clustering for image-based human-centric analysis.

  • Transformers in Human-Centric Vision Tasks: Transformers support modeling relationships between body parts and human-environment interactions in human-centric tasks.Prior approaches refine image features or aggregate part-wise features.
  • Transformers in Human-Centric Vision Tasks: Most prior methods in both categories generate vision tokens from fixed grids, which is sub-optimal for human-centric tasks.TCFormer learns token regions automatically rather than using handcrafted designs.
  • Dynamic Token Generation: Dynamic token methods alter grid number, resolution, or centers, but remain variants of grid-based token generation.Their token regions remain restricted by grid structure.
  • Token Region Flexibility: TCFormer assigns regions by semantic similarity, allowing non-adjacent regions and non-rectangular shapes in addition to flexible locations and sizes.This distinguishes its token regions from methods based on spatial proximity and rectangular grids.
  • Clustering for Feature Aggregation: Clustering methods from point clouds and graphs cannot be directly applied to image-based vision transformers, whereas TCFormer clusters tokens across stages for human-centric vision.The paper identifies this as the first use of clustering for vision token generation.

3. Method

TCFormer uses progressive token clustering to build a four-stage transformer with flexible token regions, importance-guided merging, and multi-stage aggregation that preserves details.

  • Architecture: TCFormer has four hierarchical stages, clustering-based token merge blocks between stages, and an MTA head that outputs final heatmaps.Each stage contains stacked transformer blocks, while MTA aggregates token features from all stages.
  • Transformer Block: Its transformer block reduces attention complexity with spatial reduction and captures local information using depth-wise convolution.Spatial reduction converts tokens to lower-resolution feature maps before keys and values enter multi-head attention.
  • Token Clustering: CTM clusters tokens with DPC-KNN, selects centers using local density and distance, and assigns remaining tokens to nearest centers.The clustering score is ρ_i×δ_i, where higher scores indicate greater potential to be cluster centers.
  • Feature Merging: Feature merging uses importance-weighted averaging, and the merged token region is the union of its constituent token regions.Importance scores are also added to attention weights so critical image features contribute more during merging.
  • Multi-stage Token Aggregation: MTA progressively upsamples tokens, copies merged features through recorded relationships, adds previous-stage features, and applies transformer blocks.The process ends with tokens corresponding one-to-one with high-resolution feature-map pixels, enabling reshaping without low-resolution detail loss.

4. Experiments

Experiments evaluate TCFormer across whole-body pose estimation, 3D mesh reconstruction, face alignment, image classification, and ablations. Results show strong performance across tasks, with detailed-region modeling particularly benefiting hands and feet.

  • 2D Whole-body Pose Estimation: 57.2% AP and 67.8% AR are achieved on COCO-WholeBody, exceeding state-of-the-art top-down methods such as HRNet by a large margin.The evaluation uses OKS-based AP and AR on 133 whole-body keypoints.
  • 2D Whole-body Pose Estimation: 6.2% AP higher than HRNet-w32 and 5.3% AP higher than SBL-Res152 are obtained for hand keypoint estimation.The result highlights TCFormer’s ability to capture critical details in small hand regions.
  • 3D Human Mesh Reconstruction: TCFormer outperforms most prior methods with similar structures and complexity on 3DPW and remains competitive with DSR, which uses extra dense supervision.The evaluation reports MPJPE and PA-MPJPE on 3DPW and Human3.6M.
  • 2D Face Keypoint Localization: 4.28% NME is achieved on WFLW, outperforming other state-of-the-art methods on the full test set and all six subsets.TCFormer-Light is compared using normalized mean error, including challenging pose, occlusion, illumination, and blur subsets.
  • Image Classification: 82.4% Top-1 Acc. on ImageNet-1K is competitive with state-of-the-art architectures despite image classification not being the model’s target.This result indicates that the CTM block also extracts general image features.
  • Ablation Study: Replacing CTM with strided convolution reduces performance by −3.7% AP and −3.9% AR, with larger drops for foot and hand estimation.Foot AP decreases by 13.6% and hand AP by 5.6%, compared with a 2.4% body AP decrease.
  • Ablation Study: Replacing the MTA head with a deconvolutional head reduces performance by −1.9% AP and −1.6% AR.The drops are larger for foot (−4.0% AP) and hand (−3.6% AP) than for body (−1.2% AP).
  • Qualitative Results: TCFormer allocates fine spatial tokens to detailed face and hand regions while representing complex backgrounds with few tokens.The qualitative analysis covers whole-body pose estimation, 3D mesh reconstruction, and face alignment.

5. Analysis

TCFormer’s token distribution is task-specific: clustering preserves distinct regions when their features differ, directing fine tokens toward the details required by each task.

  • Task-specific Token Allocation: Different face regions tend to remain in separate clusters because CTM groups tokens according to feature similarity.This preserves fine tokens needed to distinguish dense facial keypoints in whole-body pose estimation.
  • Task-specific Token Allocation: TCFormer changes token distribution when trained specifically for hand keypoints versus face keypoints.The task-specific distributions shown in Figure 8 match the explanation that clustering follows task-relevant features.

6. Conclusions and Limitations

TCFormer uses clustering-based token merging to preserve fine details in important regions while reducing attention to uninformative backgrounds, and performs competitively across human-centric tasks. Its main limitation is quadratic clustering complexity at large input resolutions.

  • Conclusions: TCFormer combines a CTM block with an architecture designed to preserve finer details in important regions and reduce attention to useless background information.The method is evaluated on whole-body pose estimation, human mesh recovery, and face alignment.
  • Conclusions: Experiments show significant improvement over the baseline and competitive performance across several human-centric vision tasks.The reported tasks are whole-body pose estimation, human mesh recovery, and face alignment.
  • Limitations: The KNN-DPC clustering algorithm has quadratic computational complexity in token number, limiting TCFormer speed at large input resolutions.The paper notes part-wise token clustering as a possible mitigation.

A. Detailed Settings for Image Classification

The image-classification appendix describes the training data, augmentation, model-scale parameters, and CTM placement used for detailed TCFormer settings.

  • Training Settings: Image classification experiments use ImageNet-1K, which contains 1.28 million training images, 50K validation images, and 1,000 categories.Training uses random cropping, flipping, label smoothing, Mixup, CutMix, and random erasing.
  • Model Scaling: TCFormer models are scaled for different tasks using stage-specific spatial reduction ratios, head numbers, expansion ratios, and token channel dimensions.These parameters are denoted R_i, N_i, E_i, and C_i, respectively.
  • CTM Configuration: Each CTM block contains a transformer block whose settings match the transformer blocks in the next stage.This specifies how clustering-based merging is integrated across stages.

C. 2D Whole-body Pose Estimation

TCFormer-large achieves new state-of-the-art whole-body pose performance while particularly improving estimation of small foot and hand regions. Its token distribution covers all body parts rather than concentrating tokens narrowly.

  • 2D Whole-body Pose Estimation: TCFormer-large outperforms HRNet-w48 by 1.3% AP and 1.9% AR on COCO-WholeBody.The model uses a 384 × 288 input resolution for fair comparison with methods using larger capacity and higher resolution.
  • 2D Whole-body Pose Estimation: Gains over other state-of-the-art methods are most obvious for the small foot and hand regions.These results support TCFormer's capability to capture details that occupy small areas in the input images.
  • 2D Whole-body Pose Estimation: Unlike CTM-topk, CTM produces vision tokens that cover all body parts instead of focusing most tokens on a small image area.The CTM-topk distribution can leave some body parts represented by very few tokens or merged with background tokens.

D. More Ablation Studies

The ablations show that both the clustering algorithm and CTM design are important for whole-body pose estimation. Selecting centers only by importance scores concentrates tokens excessively and causes substantial performance loss.

  • More Ablation Studies: The ablation study evaluates CTM-topk by replacing CTM blocks in TCFormer for whole-body pose estimation.CTM-topk selects cluster centers using token importance scores only.
  • More Ablation Studies: Replacing CTM with CTM-topk reduces performance by 7.0% AP and 7.0% AR.The CTM-topk model performs even worse than the baseline without CTM blocks.
  • More Ablation Studies: CTM-topk allocates most clustering centers to the highest-scoring regions, leaving middle-score regions with few or no centers.This allocation causes information loss in regions that are not among the highest-scoring areas.
  • More Ablation Studies: With CTM-topk, some body parts receive very few tokens or are merged with background tokens, whereas DPC-KNN-generated centers cover the body more broadly.The token distributions provide a visual explanation for the performance degradation caused by CTM-topk.

E. More Qualitative Results

Qualitative results show TCFormer handling detailed human features and challenging conditions across whole-body pose estimation, 3D mesh reconstruction, and face alignment. The examples indicate robustness across occlusion, appearance, illumination, background, and pose variation.

  • More Qualitative Results: TCFormer estimates hand and foot keypoints accurately in whole-body pose examples.These examples are presented as evidence of capturing small-scale details.
  • More Qualitative Results: TCFormer estimates human meshes accurately in outdoor images with varied backgrounds, illumination, and poses.The 3D mesh examples are shown on challenging outdoor images.
  • More Qualitative Results: TCFormer performs well on face-alignment cases involving occlusion, heavy makeup, rare poses, and rare illumination.The qualitative results cover whole-body pose estimation, 3D mesh reconstruction, and face alignment.

F. Visualizations about Token Distribution

Token visualizations show that TCFormer progressively adapts token allocation to task-relevant regions and semantic parts. It uses few tokens for uninformative backgrounds while preserving finer representations where the task requires detail.

  • Visualizations about Token Distribution: The visualizations cover token distributions for whole-body pose, 3D mesh reconstruction, face alignment, image classification, and task-specific hand or face estimation.Examples include results and token distributions shown in Figures A2–A9.
  • Visualizations about Token Distribution: For whole-body pose and mesh estimation, TCFormer merges background regions into very few tokens and allocates more attention to human body regions.Simple backgrounds may be merged early, while complex backgrounds require higher-level semantic features to distinguish foreground from background.
  • Visualizations about Token Distribution: On image classification, TCFormer allocates more tokens to informative regions and fewer tokens to uninformative background areas.The generated token regions align with semantic parts, extending the observed behavior beyond human-centric tasks.
  • Visualizations about Token Distribution: Token distributions are task-specific: models targeting hand keypoints and face keypoints focus their tokens on different image regions.The hand- and face-targeted models are visualized as token (hand) and token (face).
Loading 2204.08680v3…